simon/post-scarcity
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: simon:develop
Branches Tags
simon:master
simon:develop
simon:feature/all-integers-are-bignums
..
compare: simon:master
Branches Tags
simon:develop
simon:master
simon:feature/all-integers-are-bignums

No commits in common. "develop" and "master" have entirely different histories.
develop ... master

49 changed files with 328 additions and 2348 deletions
Show stats Expand all files Collapse all files

57
Doxyfile
View file

@ -76,7 +76,7 @@ CREATE_SUBDIRS = NO
# U+3044.
# The default value is: NO.
ALLOW_UNICODE_NAMES = YES
ALLOW_UNICODE_NAMES = NO
# The OUTPUT_LANGUAGE tag is used to specify the language in which all
# documentation generated by doxygen is written. Doxygen will use this
@ -310,7 +310,7 @@ MARKDOWN_SUPPORT = YES
# Minimum value: 0, maximum value: 99, default value: 0.
# This tag requires that the tag MARKDOWN_SUPPORT is set to YES.
TOC_INCLUDE_HEADINGS = 5
TOC_INCLUDE_HEADINGS = 0
# When enabled doxygen tries to link words that correspond to documented
# classes, or namespaces to their corresponding documentation. Such a link can
@ -790,7 +790,7 @@ WARN_LOGFILE = doxy.log
# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
# Note: If this tag is empty the current directory is searched.
INPUT = src docs lisp
INPUT = src
# This tag can be used to specify the character encoding of the source files
# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
@ -816,11 +816,50 @@ INPUT_ENCODING = UTF-8
# *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf and *.qsf.
FILE_PATTERNS = *.c \
*.cc \
*.cxx \
*.cpp \
*.c++ \
*.java \
*.ii \
*.ixx \
*.ipp \
*.i++ \
*.inl \
*.idl \
*.ddl \
*.odl \
*.h \
*.lisp \
*.hh \
*.hxx \
*.hpp \
*.h++ \
*.cs \
*.d \
*.php \
*.php4 \
*.php5 \
*.phtml \
*.inc \
*.m \
*.markdown \
*.md
*.md \
*.mm \
*.dox \
*.py \
*.pyw \
*.f90 \
*.f95 \
*.f03 \
*.f08 \
*.f \
*.for \
*.tcl \
*.vhd \
*.vhdl \
*.ucf \
*.qsf
# The RECURSIVE tag can be used to specify whether or not subdirectories should
# be searched for input files as well.
# The default value is: NO.
@ -943,7 +982,7 @@ FILTER_SOURCE_PATTERNS =
# (index.html). This can be useful if you have a project on for instance GitHub
# and want to reuse the introduction page also for the doxygen output.
USE_MDFILE_AS_MAINPAGE = docs/Home.md
USE_MDFILE_AS_MAINPAGE =
#---------------------------------------------------------------------------
# Configuration options related to source browsing
@ -1439,7 +1478,7 @@ DISABLE_INDEX = NO
# The default value is: NO.
# This tag requires that the tag GENERATE_HTML is set to YES.
GENERATE_TREEVIEW = YES
GENERATE_TREEVIEW = NO
# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that
# doxygen will group on one line in the generated HTML documentation.
@ -1494,7 +1533,7 @@ FORMULA_TRANSPARENT = YES
# The default value is: NO.
# This tag requires that the tag GENERATE_HTML is set to YES.
USE_MATHJAX = YES
USE_MATHJAX = NO
# When MathJax is enabled you can set the default output format to be used for
# the MathJax output. See the MathJax site (see:

6
Makefile
View file

@ -3,7 +3,7 @@ SRC_DIRS ?= ./src
SRCS := $(shell find $(SRC_DIRS) -name *.cpp -or -name *.c -or -name *.s)
HDRS := $(shell find $(SRC_DIRS) -name *.h)
OBJS := $(addsuffix .o,$(basename $(SRCS)))
OBJS := $(addsuffix .o,$(basename $(SRCS)))
DEPS := $(OBJS:.o=.d)
TESTS := $(shell find unit-tests -name *.sh)
@ -21,8 +21,6 @@ DEBUGFLAGS := -g3
all: $(TARGET)
Debug: $(TARGET)
$(TARGET): $(OBJS) Makefile
$(CC) $(DEBUGFLAGS) $(LDFLAGS) $(OBJS) -o $@ $(LDFLAGS) $(LOADLIBES) $(LDLIBS)
@ -36,7 +34,7 @@ else
indent $(INDENT_FLAGS) $(SRCS) $(HDRS)
endif
test: $(TESTS) Makefile $(TARGET)
test: $(OBJS) $(TESTS) Makefile
bash ./unit-tests.sh
.PHONY: clean

130
docs/Access-control.md
View file

@ -1,130 +0,0 @@
# Access control
*Not yet implemented*
_Note that a number of details not yet finalised are used in examples in this note. There must be some mechanism for creating fully qualified and partially qualified hierarchical names, but I haven't finalised it yet. In this note I've assumed that the portions of an hierarchical name are separated by periods ('.'); that fully qualified names start with a quote mark; and that where a name doesn't start with a quote mark, the first portion of it is evaluated in the current environment and its value assumed to be a fully qualified equivalent. All of these details may change._
In a multi-user environment, access control is necessary in order for a user to be able to protect an item of data from being seen by someone who isn't authorised to see it. But actually, in a world of immutable data, it's less necessary than you might think. As explained in my note on [Memory, threads and communication](https://www.journeyman.cc/blog/posts-output/2017-01-08-post-scarcity-memory-threads-and-communication/), if there's strict immutability, and all user processes spawn from a common root process, then no user can see into any other user's data space anyway.
But that makes collaboration and communication impossible, so I've proposed namespaces be mutable. So the value of a name in a [namespace](Namespace.html) will be a data item and inevitably that data item will be in some user's data space. So we do need an access control list on each data item.
## Initial thoughts
My initial plan was that the access control list would have the following structure:
1. **NIL** would mean only the owner could read it;
2. **T** would mean that anyone could read it;
3. A list of user objects and group objects, any of whom could read it.
This does not work. We don't know who the owner of a cell is; in the current design we don't store that information, and I cannot see merit in storing that information. So **NIL** cannot mean 'only the owner could read it'. It essentially means 'no-one can read it' (or possibly 'only system processes can read it, which is potentially useful).
Worse though, if the list points to immutable user and group objects, then if a new user is added to the system after the object was created, they can never be added to the access control list.
## Write access
As most data is immutable, there's no need for write access lists. If it exists, you can't write it, period. You can make a modified copy, but you can't modify the original. So most data objects don't need a write access list.
A sort-of minor exception to this is write streams. If you have normal access to a write stream, gatekept by the normal access lists, you can write to the stream; what you can't do is change where the stream points to. As you can't read from a write stream, there's still only one access list needed.
However, if (some) [namespaces](Namespace.html) are mutable - and I believe some must be - then a namespace does need a write access list, in addition to its (normal) read access list. The structure of a write access list will be the same as of a read access list.
### Modifying write access lists on mutable namespaces
If mutable namespaces have write access lists, then someone has to be able to manage the content of those write access lists - including modify them. Of course, data being immutable, 'modify' here really means replace. Who has authority to modify the access control list of a mutable namespace? The simplest thing would be for mutable namespaces to have an extra key, '**write-access**', which pointed to the write access list. Then any user with write access to the namespace could modify the write access list. That may be undesirable and needs further thought, but any other solution is going to be complex.
## Execute access
I don't see the need for separate read and execute access lists. This is possibly slightly affected by whether the system can run interpreted code, compiled code, or both. If it can run interpreted code, then having read access to the source is equivalent to having execute access, unless there is a separate execute access list (which I don't want). Thus, a user able to edit a system function would also be able to execute it - but as themselves, not as system, so it would not be able to call any further system functions it depended on, unless the user also had read access to them. Note that, of course, in order to put the new version of the function into the system namespace, to make it the version which will be called by other processes, the user would need write access to the system namespace.
However, it's really hard to make the semantics of interpreted code identical to compiled code, and compilation is no longer such a big deal on modern fast processors. So I don't see the necessity of being able to run interpretted code; it's easier if source and executable are different objects, and, if they're different objects, they can have different access lists. So having access to the source doesn't necessarily means having access to the executable, and vice versa.
If only compiled code can be executed, then it seems to me that having access to the compiled code means one can execute it, and still there's only one access list needed.
## Read access
Thus the default access list is the read access list; every cell has an access list. What do its possible values mean?
1. **T** everyone can read this;
2. An executable function of two arguments: the current user can read the cell if the function, when passed as arguments the current user and the cell to which access is requested, returns **T** or a list of names as below, and the user is present on that list;
3. A list of names: true if the value of one of those names is the user object of the current user, or is a group which contains the user object of the current user.
If there's anything on the list which isn't a name it's ignored. Any value of the access list which isn't **T**, an executable function, of a list of names is problematic; we either have to treat it as **T** (everyone) or as **NIL** (either no-one or system-only). We should probably flag an error if an attempt is made to create a cell with an invalid access list. Access control list cells also clearly have their own access control lists; there is a potential for very deep recursion and consequently poor performance here, so it will be desirable to keep such access control lists short or just **T**. Obviously, if you can't read an access control list you can't read the cell that it guards.
## If data is immutable, how is an access control list set?
My idea of this is that there will be a priviliged name which is bound in the environment of each user; each user will have their own binding for this name, and, furthermore, they can change the binding of the name in their environment. For now I propose that this name shall be **friends**. The value of **friends** should be an access list as defined above. The access control list of any cell is the value that **friends** had in the environment in the environment in which it was created, at the time it was created.
## Managing access control
The `with` function can be used to make this easier:
```
(with ((*friends* . list-or-t-or-executable)) s-exprs...)
```
Creates a new environment in which **friends** is bound to the value of **list-or-t-or-executable**, and within that environment evaluates the specified **s-exprs**. Any cells created during that evaluation will obviously have **list-or-t-or-executable** as their access control. Returns the value of executing the last **s-expr**.
### (get-access-control s-expr)
Returns the access control list of the object which is the value of the **s-expr**.
### Typical use cases
Suppose I want to compile a function **foo** which will be executable by all my current friends and additionally the group **foo-users**:
```
(with-open-access-control
(cons ::system:groups:foo-users *friends*)
(rebind! ::system:users:simon:functions:foo (compile foo))
```
_Here **rebind!** creates a new binding for the name **foo** in the namespace **::system:users:simon:functions**, whether or not that name was previously bound there. Analogous to the Clojure function **swap!**_
Suppose I want to compile a function **bar** which will be executable by exactly the same people as **foo**:
(with-access-control
(get-access-control 'system.users.simon.exec.foo)
(rebind! 'system.users.simon.exec.bar (compile bar))
Suppose I want to do some work which is secret, visible only to me and not to my normal friends:
```
(with ((*friends* . (list current-user)))
(repl))
```
(or, obviously,
```
(with ((*friends* current-user))
(repl))
```
which is in practice exactly the same)
_Here **repl** starts a new read-eval-print loop in the modified environment - I suspect this is a common use case._
Suppose I want to permanently add Anne and Bill to my normal friends:
```
(rebind! *environment*:*friends* (cons ::system:users:anne (cons ::system:users:bill *friends*)))
```
_Here I'm presuming that `*environment*` is bound to the value of `::system:users:simon:environment`, and that unqualified names are searched for first in my own environment._
Suppose I want everyone to be able to play a game, but only outside working hours; and for my friends to be able to play it additionally at lunchtime:
```
(with ((*friends*
(compile
(lambda (user cell)
(let ((time . (now)))
(cond
((< time 09:00) T)
((> time 17:00) T)
((and (> time 12:30)(< time 13:30)) *friends*)
(T NIL)))))))
(rebind! ::system:users:simon:functions:excellent-game (compile excellent-game)))
```
## Summary
Security is hard to do well, and all this requires careful further thought, in part because the proposed post-scarcity environment is so unlike any existing software environment.

141
docs/Cons-space.md
View file

@ -1,141 +0,0 @@
# Cons space
*See [cons_space_object.h](consspaceobject_8h.html), [cons_page.h](conspage_8h.html).*
Cons space is a space which contains cons cells, and other objects whose memory representation fits into the footprint of a cons cell. A cons cell comprises:
+-----+-------+------+----------------+--------------+--------------+
| header | payload |
+ +--------------+--------------+
| | car | cdr |
+-----+-------+------+----------------+--------------+--------------+
| tag | count | mark | access-control | cons-pointer | cons-pointer |
+-----+-------+------+----------------+--------------+--------------+
The size of each of these components needs to be fixed in a running system. Traditional Lisps have made each as small as possible, but part of the point of thinking about post-scarcity software is to get away from thinking parsimoniously. While obviously we do still have finite store and at some stage in development it will be time to ask what are the sensible compromises to make, that time is not now. To get up and running, and to make debugging of memory sllocation easy, we'll be expansive.
## Header
Every cons space object has a header and each header shall be the same size with the same subdivisions.
### Tag
The tag identifies the type of the cons space object. There must be at least sufficient tag values for all the possible types of cons space objects. Actually the number of potential types of cons space objects is quite low, but at present I don't know how many I'll need. Sixteen values (4 bits) may be enough, but for the time being we'll reserve 32 bits. We'll think of these bits both as an unsigned 32 bit integer, and as a string of four ASCII characters; that is, we'll assign for now only numeric values which, if considered as an ASCII string, result in useful mnemonics. This will make memory dumps easy to read, which will aid in debugging memory allocation.
### Count
Either the count or the mark are redundant. The count is for reference counting garbage collection; the mark is for mark-and-sweep (including generational mark-and-sweep) garbage collection. Ultimately we probably need only one of these; conventional wisdom is that generational mark and sweep will win. But I want to banchmark both systems and see how they perform, so for now we'll have both.
A reference count counts how many other objects point to this object. When the reference count decrements to zero, the object may safely be garbage collected. However, when a reference count can no longer be safely incremented, neither can it ever be safely decremented. Suppose we had three bits - eight values including zero, 0...7 - for the reference count. Suppose six other objects point to this object, so the reference count is 6. Now suppose one of those objects is freed, so no longer points to this object. Our reference count is decremented to 5, and that's OK.
But, suppose seven objects already point to this object; our reference count is now 7. If an eigth object is created which points to this object, we cannot increment the reference count because we no longer have bits to store the incremented value. So we have to leave it at 7. Now, suppose another object which points to this object is freed: do we decrement the reference counter? No: we can't, because we can't know whether the actual number of objects which point to it is seven, or eight, or one hundred.
Consequently, for any size of reference counter, when it hits its maximum value it can no longer be decremented, and consequently a reference counting garbage collector can no longer free that object - ever. It is possible to write a hybrid reference-counting/mark-and-sweep garbage collector, but that is both expensive and complicated. We need a size of reference count which will very, very rarely overflow in practice. That's probably still quite small, but I'm proposing to reserve 24 bits (16,777,216 values) (in fact the current implementation reserves 32 bits - see [consspaceobject.h](https://github.com/simon-brooke/post-scarcity/blob/master/src/consspaceobject.h)).
### Mark
A mark and sweep garbage collector actually only needs one mark bit, but for now it will sit in the same space as the reference count, since we're only using one or other, never both.
### Access control
Access control is a [cons pointer](cons pointer.html), see below; and is consequently the size of a cons pointer, which is presently 64 bits. An access control value of NIL means only system processes may access the cell; an access control value of TRUE means any user can access the cell; otherwise, the access control pointer points to the first cons cell of a list of allowed users/groups. The access control list is thus an ordinary list in ordinary cons space, and cells in an access control list can have access control lists of their own. As cons cells are immutable, infinite recursion is impossible; but it is nevertheless probably a good thing if access control list cells normally have an access control list of either TRUE or NIL.
### Car, Cdr: Cons pointers
A [cons pointer](cons pointer.html) is simply a pointer to a cons cell, and the simplest way to implement this is exactly as the memory address of the cons cell.
We have a fixed size vector of total memory, which we address in eight bit words (bytes) because that's the current convention. Our cons cell size is 32 bytes. So 31/32 of the possible values of a cons pointer are wasted - there cannot be a valid cons cell at that address. Also, our total memory must be divided between cons space, vector space and stack (actually stack could be implemented in either cons space or vector space, and ultimately may end up being implemented in cons space, but that's a highly non-trivial detail which will be addressed much later). In practice it's likely that less than half of the total memory available will be devoted to cons space. So 63/64 of the possible values of a cons pointer are wasted.
Is there a better way? Yes, there is, but as in all engineering matters it's a trade off.
One of the things I absolutely hate about modern computers is their tendency to run out of one 'sort' of memory while there is actually plenty of memory free. For example, it's childishly easy to run any JVM program out of stack space, because the JVM on startup reserves a fixed size block of memory for stack, and cannot extend this block. When it's exhausted, execution halts, and you've had your chips. There is no recovery.
That was acceptable when the JVM was a special purpose platform for developing software for small embedded devices, which is what it was originally designed for. But it's one of the compromises the JVM makes in order to work well on small embedded devices which is completely unacceptable for post-scarcity computing. And we won't accept it.
But be that as it may, we don't know at system initialisation time how much memory to reserve for cons space, and how much for vector space ('the heap'). If we reserve too much for cons space, we may run out of vector space while there's still cons space free, and vice versa. So we'll reserve cons space in units: [cons pages](cons pages.html). If our cons pointers are absolute memory addresses, then it becomes very expensive to move a cons page in memory, because all the pointers in the whole system to any cell on the page need to be updated.
(**NOTE**: As my thinking has developed, I'm now envisaging one cons page per compute node, which means that on each node the division between cons space and vector space will have to be fixed)
If, however, we divide our cons pointer into two elements, a page number and a page offset. Suppose we have 40 bits of page number (up to 1,099,511,627,776 - one trillion - pages) and 24 bits of page offset (up to 16,777,216 cons cells on a page), then suddenly we can address not 2^64 bytes of memory in cons space, but 32x2^64. Furthermore, this cons space addressing is now independent of vector space addressing, allowing even further address space.
Obviously it also means that to fetch any cell, the processor has to first fetch the address of the page, then compute the address of the offset of the cell in the page, then fetch the cell at the computed address, making three processor cycles instead of just one. We're post-scarcity: at this stage, we don't worry about such things. The time to worry about run-time performance is far beyond version 0.
So our cons cell is now 32 bytes, 256 bits:
+-----+-------+------+----------------+--------------+--------------+
| header | payload |
+ +--------------+--------------+
| | car | cdr |
+-----+-------+------+----------------+--------------+--------------+
| 0 | 32 | 56 | 64 | 128 | 192 ...255 |
| tag | count | mark | access-control | cons-pointer | cons-pointer |
+-----+-------+------+----------------+--------------+--------------+
## Types of cons space object
This is a non-exhaustive list of types of things which may be stored in cons space; each has a memory representation which is 128 bits or less, and will thus fit in the memory footprint of a cons cell. There will be others I have not yet thought of, but this is enough to get us started.
### CONS
A cons cell. The tag value of a CONS cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'CONS'. The count of a CONS cell is always non-zero. The mark is up to the garbage collector. The Car of a CONS cell is a pointer to another cons-space object, or NIL (address zero).
### FREE
An unassigned cons cell. The tag value of a FREE cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'FREE'. The count of a FREE cell is always zero. The mark of a free cell is always zero. The access control value of a FREE cell is always NIL. The Car of a FREE cell is always NIL (address zero). The Cdr of a FREE cell is a cons-pointer to the next FREE cell (the [free list](free list.html) pointer).
### INTR
An integer; possibly an integer which isn't a big integer. The tag value of a INTR cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'INTR'. The count of a INTR cell is always non-zero. The mark is up to the garbage collector.
There's fundamentally two ways to do this; one is we store up to 128 bit signed integers in the payload of an INTR cell, and have some other tag for an integer ('[bignum](bignum.html)') which overflows 128 bits and must thus be stored in another data structure; or else we treat one bit as a 'bignum' flag. If the bignum flag is clear we treat the remaining 127 bits as an unsigned 127 bit integer; if set, we treat the low 64 bits of the value as a cons pointer to the data structure which represents the bignum.
### NIL
The canonical empty list. May not actually exist at all: the cell-pointer whose value is zero is deemed to point to the canonical empty list. However, if zero is a valid cell-pointer, the cell at pointer zero will be initialised with the tag "NIL " (i.e. the 32 bit unsigned integer which, when considered as an ASCII string, reads "NIL "). The count of the NIL cell is the maximum reference count value - that is, it can never be garbage collected. The mark is always 1 - that is, it can never be garbage collected. The access control value is TRUE - any user can read NIL. The payload is zero.
### READ
A stream open for reading. The tag value of a READ cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'READ'. The count of a READ cell is always non-zero. The mark is up to the garbage collector.
I'm not yet certain what the payload of a READ cell is; it is implementation dependent and, at least in version zero, will probably be a file handle from the underlying system.
### REAL
A real number. The tag value of a REAL cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'REAL'. The count of a REAL cell is always non-zero. The mark is up to the garbage collector. The payload is a IEEE 754R 128-bit floating point number.
### STRG
A string. The tag value of a STRG cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'STRG'. The count of a STRG cell is always non-zero. The mark is up to the garbage collector. The Car of an STRG cell contains a single UTF character. The Cdr of an STRG cell contains a cons-pointer to the remainder of the string, or NIL if this is the end of the string.
Note that in this definition a string is not an atom, which is probably right. But we also at this stage don't have an idea of a [symbol](Interning-strings.html). Very likely we'll end up with the idea that a string which is bound to a value in a namespace is for our purposes a symbol.
Note, however, that there's a risk that we might have two instances of strings comprising identical characters in identical order, one of which was bound in a namespace and one of which wasn't; string equality is something to worry about.
### TIME
At nanosecond resolution (if I've done my arithmetic right), 128 bits will represent a span of 1 x 10<sup>22</sup> years, or much longer than from the big bang to the [estimated date of fuel exhaustion of all stars](https://en.wikipedia.org/wiki/Timeline_of_the_far_future). So I think I'll arbitrarily set an epoch 14Bn years before the UNIX epoch and go with that. The time will be unsigned - there is no time before the big bang.
### TRUE
The canonical true value. May not actually exist at all: the cell-pointer whose value is one is deemed to point to the canonical true value. However, if one is a valid cell-pointer, the cell at pointer zero will be initialised with the tag "TRUE" (i.e. the 32 bit unsigned integer which, when considered as an ASCII string, reads "TRUE"). The count of the TRUE cell is the maximum reference count value - that is, it can never be garbage collected. The mark is always 1 - that is, it can never be garbage collected. The access control value is TRUE: any user can read the canonical true value. The payload is zero.
### VECP
A pointer into vector space. The tag value of a VECP cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'VECP'. The count of a VECP cell is always non-zero. The mark is up to the garbage collector. The payload is the a pointer to a vector space object. On systems with an address bus up to 128 bits wide, it's simply the address of the vector; on systems with an address bus wider than 128 bits, it's probably an offset into an indirection table, but that really is a problem for another day.
As an alternate implementation on hardware with a 64 bit address bus, it might be sensible to have the Car of the VECP cell simply the memory address of the vector, and the Cdr a pointer to the next VECP cell, maintained automatically in the same way that a [free list](Free-list.html) is maintained. This way we automatically hold a list of all live vector space objects, which would help in garbage collecting vector space.
Every object in vector space shall have exactly one VECP cell in cons space which refers to it. Every other object which wished to hold a reference to that object shall hold a cons pointer to VECP cell that points to the object. Each object in vector space shall hold a backpointer to the VECP cell which points to it. This means that if vector space needs to be shuffled in order to free memory, for each object which is moved only one pointer need be updated.
When the reference count of a VECP cell is decremented to zero, the backpointer on the vector to which it points will be set to NIL (zero), marking it as available for garbage collection.
### WRIT
A stream open for writing. The tag value of a WRIT cell is that unsigned 32 bit integer which, when considered as an ASCII string, reads 'WRIT'. The count of a WRIT cell is always non-zero. The mark is up to the garbage collector.
I'm not yet certain what the payload of a WRIT cell is; it is implementation dependent and, at least in version zero, will probably be a file handle from the underlying system.
## Cons pages
Cons cells will be initialised in cons pages. A cons page is a fixed size array of cons cells. Each cell is initialised as FREE, and each cell, as it is initialised, is linked onto the front of the system [free list](Free-list.html). Cons pages will exist in [vector space](Vector-space.html), and consequently each cons page will have a vector space header.

95
docs/Core-functions.md
View file

@ -1,95 +0,0 @@
# Core functions
*See [ops/lispops.h](lispops_8h.html).*
In the specifications that follow, a word in all upper case refers to a tag value, defined on either the [cons space](Cons-space.html) or the [vector space](Vector-space.html) page.
# (and args...)
Public. Takes an arbitrary number of arguments. Returns true if all are readable by the current user and evaluate to non-NIL. _Note that evaluation of args may be parallelised across a number of processors, so you *cannot* use this for flow control._
# (atom? arg)
Public. Takes one argument. Returns TRUE if that argument is neither a CONS cell, a VECP, nor a STRG cell, else NIL.
# (append args...)
Public. Takes an arbitrary number of arguments, which should either all be CONS cells or all STRG cells. In either case returns a concatenation of all those arguments readable by the current user.
# (assoc key store)
Public. Takes two arguments, a key and a store. The store may either be a CONS forming the head of a list formatted as an [assoc list](Hybrid-assoc-lists.html), or else a VECP pointing to a HASH. If the key is readable by the current user, returns the value associated with that key in the store, if it exists and is readable by the current user, else NIL.
# (car arg)
Public. Takes one argument. If that argument is a CONS cell and is readable by the current user, returns the value indicated by the first pointer of that cell; if the argument is an STRG and is readable by the user, a CHAR representing the first character in the string; else NIL.
# (cdr arg)
Public. Takes one argument. If that argument is a CONS or STRG cell and is readable by the current user, returns the value indicated by the second pointer of that cell, else NIL.
# (cond args...)
Public. Takes an arbitrary number of arguments each of which are lists. The arguments are examined in turn until the first element of an argument evaluates to non-nil; then each of the remaining elements of that argument are evaluated in turn and the value of the last element returned. If no argument has a first element which evaluates to true, returns NIL. _Note: this is explicit flow control and clauses will not be evaluated in parallel._
# (cons a d)
Public. Takes two arguments, A, D. Returns a newly allocated cons cell whose first pointer points to A and whose second points to D.
# (eq? a b)
Public. Takes two arguments, A, B. Returns TRUE if both are readable by the current user and are the same cons space object (i.e. pointer equality), else NIL.
# (eval arg)
Public. Takes one argument.
* if that argument is not readable by the current user, returns NIL.
* if that argument is a CONS, returns the result of
(apply (car arg) (cdr arg))
* if that argument is an INTR, NIL, REAL, STRG, TRUE or VECP, returns the argument.
* if that argument is a READ or a WRIT, _probably_ returns the argument but I'm not yet certain.
# (intern str), (intern str namespace)
Public.
* If one argument, being a STRG readable by the current user, interns that string as a symbol in the current namespace (by binding it to a special symbol *sys-intern*, which has its access control set NIL).
* if two arguments, being a STRG and a VECP pointing to a HASH, interns that string in the specified hash.
_Note: I'm not sure what happens if the STRG is already bound in the HASH. A normal everyday HASH ought to be immutable, but namespaces can't be immutable or else we cannot create new stuff._
# (lambda args forms...)
Public. Takes an arbitrary number of arguments. Considers the first argument ('args') as a set of formal parameters, and returns a function composed of the forms with those parameters bound. Where I say 'returns a function', this is in initial prototyping probably an interpreted function (i.e. a code tree implemented as an S-expression), but in a usable version will mean a VECP (see [cons space](Cons-space.html#VECP)) pointing to an EXEC (see [vector space#EXEC](Vector-space.html#EXEC)) vector.
# (nil? arg)
Public. Takes one argument. Returns TRUE if the argument is NIL. May also return TRUE if the argument is not readable by the current user (on the basis that what you're not entitled to read should appear not to exist) but this needs more thought.
# (not arg)
Public. Takes one argument. Returns TRUE if that argument is NIL, else NIL. _Note: Not sure what happens when the argument is not NIL but not readable by the current user. If we return NIL, as we usually do for an unreadable argument, then that's a clue that the object exists but is not readable. Generally, when an object is not readable, it appears as though it doesn't exist._
# (number? arg)
Public. Takes one argument. Returns TRUE if the argument is readable by the current user and is an INTR, REAL, or some other sort of number I haven't specified yet.
# (or args...)
Public. Takes an arbitrary number of arguments. Returns TRUE if at least one argument is readable by the current user and evaluates to non-NIL. _Note that evaluation of args may be parallelised across a number of processors, so you *cannot* use this for flow control._
# (print arg write-stream)
Public. Takes two arguments, the second of which must be a WRIT that the current user has access to. Writes the canonical printed form of the first argument to the second argument.
# (quote arg)
Public. Takes one argument. Returns that argument, protecting it from evaluation.
# (read arg)
Public. Takes one argument. If that argument is either an STRG or a READ, parses successive characters from that argument to construct an S-expression in the current environment and returns it.
# (type arg)
Public. Takes one argument. If that argument is readable by the current user, returns a string interned in the *core.types* namespace representing the tag value of the argument, unless the argument is a VECP in which case the value returned represents the tag value of the [vector space](Vector-space.html) object indicated by the VECP.

8
docs/Free-list.md
View file

@ -1,8 +0,0 @@
# Free list
A free list is a list of FREE cells consed together. When a cell is deallocated, it is consed onto the front of the free list, and the system free-list pointer is updated to point to it. A cell is allocated by popping the front cell off the free list.
If we attempt to allocate a new cell and the free list is empty, we allocate a new code page, cons all its cells onto the free list, and then pop the front cell off it.
However, because we wish to localise volatility in memory in order to make maintaining a consistent backup image easier, it may be worth maintaining a separate free list for each page, and allocating cells not from the front of the active free list but from the free list of the currently most active page.

158
docs/Hashing-structure-writ-large.md
View file

@ -1,158 +0,0 @@
# Hashing structure writ large
In Lisp, there's an expectation that any object may act as a key in a hash table. What that means, in practice, is that if a list
```lisp
'(foo bar ban)
```
is a key, and we pass a list
```lisp
'(foo bar ban)
```
as a query, we ought to get the value to which the first instance of `'(foo bar ban)' was bound, even if the two instances are not the same instance. Which means we have to compute the hash value by exploring the whole structure, no matter how deep and convoluted it may be.
## The cost of this
The cost of this, in the [post scarcity software environment](https://git.journeyman.cc/simon/post-scarcity) is potentially enormous: is potentially a blocker which could bring the whole project down. The post-scarcity architecture as currently conceived allows for 2<sup>64</sup> cons cells. Take the most obvious example of a wholly linear, non-branching structure, a string: it would be perverse, but possible to have a single string which occupied the entire address space.
But to be less perverse, the text of English Wictionary is 3.9 billion words, so it's reasonable to assume that the text of the Encyclopaedia Brittanica is of the same order of magnitude. There are, on average, 4.7 characters in an English word, plus slightly more than one for punctuation, so we can round that up to six. So the string required to store the text of the Encyclopedia Brittanica would be approximately 24 billion characters long; and storing that in a string would not, in the context of the post-scarcity software environment, be utterly perverse.
But the cost of hashing it would be enormous: would be even greater in the hypercube architecture of the proposed [post scarcity hardware](Post-scarcity-hardware.html) than on one with a [von Neumann architecture](https://en.wikipedia.org/wiki/Von_Neumann_architecture), since we cannot even split the string into chunks to take advantage of parallelism (since the string will almost certainly be non-contiguous in memory), so the hash has to be computed in a single thread, and also since all the cells of the string -- inevitably the majority -- not native to the memory map of the processor node calculating the hash have to be fetched hop-de-hop across the lattice of the hypercube, with each hop costing a minimum of six clock ticks (about 260 clock ticks over a serial link).
A fully populated post scarcity hardware implementation -- i.e. one large enough to contain such perverse strings -- would be a hypercube of side 1625, which means the longest path between any pair of nodes is 812 hops, which means the average path is 406 hops. But, one in every four hops, if the machine is built as I currently conceive it, is a serial link. So the average cost of fetching a datum from an arbitrary node is (6 x 3 x (406 / 4)) + (260 x (406 / 4)), which is to say, roughly, 28,700 clock ticks.
So to fetch the whole string takes about 30,000 x 24,000,000,000, or 720,000,000,000,000 clock ticks, which, assuming a 3GHz clock, is about quarter of a million seconds, or three days.
To make matters worse, suppose we now stored the hash value in a hash table as the value of that string so as to avoid having to compute it again, we could then not ever garbage collect the string, since that hash table would contain a pointer to it.
So clearly, hashing structures when required in the post scarcity software environment just will not work.
## Finding a solution
Necessarily, most data structures in the post scarcity software environment must be immutable, because most of the time we will be operating on copies of them in compute nodes remote from the node to which they are native. Thus, we can compute a hash value when the structure is first created, and cache it on the structure itself.
This option first occurred to me in the special case of string-like-things (strings, atoms, keywords). Because a wide (32 bit UTF) character sits in 32 bits of memory, and a string cell has to sit in the memory footprint of a cons cell, my string payload had 32 unused bits:
```c
/**
* payload of a string cell. At least at first, only one UTF character will
* be stored in each cell. The doctrine that 'a symbol is just a string'
* didn't work; however, the payload of a symbol cell is identical to the
* payload of a string cell.
*/
struct string_payload {
wint_t character; /* the actual character stored in this cell */
uint32_t padding; /* unused padding to word-align the cdr */
struct cons_pointer cdr;
};
```
So it was straightforward to reassign that unused `padding` as a cache for the hash value:
```c
/**
* payload of a string cell. At least at first, only one UTF character will
* be stored in each cell. The doctrine that 'a symbol is just a string'
* didn't work; however, the payload of a symbol or keyword cell is identical
* to the payload of a string cell, except that a keyword may store a hash
* of its own value in the padding.
*/
struct string_payload {
/** the actual character stored in this cell */
wint_t character;
/** a hash of the string value, computed at store time. */
uint32_t hash;
/** the remainder of the string following this character. */
struct cons_pointer cdr;
};
```
But new items can be consed onto the front of lists, and that means, in practice, new characters can be consed onto the front of strings, too. What this means is that
```lisp
(hash "the quick brown fox jumped over the lazy dog")
```
and
```lisp
(hash (append "the" "quick brown fox jumped over the lazy dog"))
```
must return the same value. But, obviously, we don't want to have to walk the whole structure to compute the hash, because we cannot know in principle, when passed a string, whether or not it is extremely long.
The answer that occurred to me, for strings, is as follows:
```c
/**
* Return a hash value for this string like thing.
*
* What's important here is that two strings with the same characters in the
* same order should have the same hash value, even if one was created using
* `"foobar"` and the other by `(append "foo" "bar")`. I *think* this function
* has that property. I doubt that it's the most efficient hash function to
* have that property.
*
* returns 0 for things which are not string like.
*/
uint32_t calculate_hash(wint_t c, struct cons_pointer ptr)
{
struct cons_space_object *cell = &pointer2cell(ptr);
uint32_t result = 0;
switch (cell->tag.value)
{
case KEYTV:
case STRINGTV:
case SYMBOLTV:
if (nilp(ptr))
{
result = (uint32_t)c;
}
else
{
result = ((uint32_t)c *
cell->payload.string.hash) &
0xffffffff;
}
}
return result;
}
```
That is to say, the hash value is the least significant 32 bits of the product of multiplying the hash of the tail of the string by the character code of the character added. This means we have a very small fixed cost -- one 32 bit integer multiplication -- every time a character is added to a string, rather than an enormous cost every time the hash value of a string is required. I *believe* this is a good trade-off, just as I believe using reference counting rather than mark-and-sweep for garbage collection is a good trade-off: it's better, in my opinion, to have steady is slightly slower than optimal performance from the machine than for it to intermittently lock up for no apparent reason.
In any case, what this means is that getting the hash value -- by the standard hashing function -- of a string cell is absurdly cheap: it's just a bit mask on the cell.
Is this a good hash function? Probably not. A hash function should ideally distribute arbitrary values pretty evenly between hash buckets, and this one is probably biased. Perhaps at some stage someone will propose a better one. But in practice, I believe that this will do for now. It is, after all, extremely cheap.
## To generalise, or not to generalise?
Certainly in Clojure practice, keys in hash maps are almost always 'keywords', a particular variety of string-like-thing. Nevertheless, Clojure, like Common Lisp (and many, perhaps all, other lisps) allows, in principal, that any value can be used as a key in a hash map.
The hack outlined above of using 32 bits of previously unused space in the string payload works because there were 32 unused bits in the string payload. While a hash function with similar properties can be imagined for cons cells, there are not currently any unused bits in a cons payload. To add a hash value would require adding more bits to every cons space object.
Our current cons space object is 256 bits:
```
+-----+--------------+----------------+--------------+--------------+
| header | payload |
+ +--------------+--------------+
| | car | cdr |
+-----+--------------+----------------+--------------+--------------+
| tag | count / mark | access-control | cons-pointer | cons-pointer |
+-----+--------------+----------------+--------------+--------------+
| 32 | 32 | 64 | 64 | 64 |
+-----+--------------+----------------+--------------+--------------+
```
This means that cells are word aligned in a 64 bit address space, and a perfect fit for a reasonably predictable future general purpose processor architecture with a 256 bit address bus. If we're now going to add more bits for hash cache, it would be perverse to add less than 64 bits because we'd lose word alignment; and it's not particularly likely, I think, that we'll see future processors with a 320 bit address bus.
It would be possible to steal some bits out of either the tag, the count, or both -- 32 bits for the tag, in particular, is absurdly generous, but it does help greatly in debugging.
For now, I'm of a mind to cache hash values only for string-like-things, and, if users want to use other types of values as keys in hash maps, use access-time hashing functions. After all, this works well enough in Common Lisp.

30
docs/Home.md
View file

@ -1,30 +0,0 @@
# Post Scarcity Software Environment: general documentation
Work towards the implementation of a software system like that described in [Post Scarcity Software](https://www.journeyman.cc/blog/posts-output/2006-02-20-postscarcity-software/).
## Note on canonicity
*Originally most of this documentation was on a wiki attached to the [GitHub project](https://github.com/simon-brooke/post-scarcity); when that was transferred to [my own foregejo instance](https://git.journeyman.cc/simon/post-scarcity) the wiki was copied. However, it's more convenient to keep documentation in the project with the source files, and version controlled in the same Git repository. So while both wikis still exist, they should no longer be considered canonical. The canonical version is in `/docs`, and is incorporated by [Doxygen](https://www.doxygen.nl/) into the generated documentation &mdash; which is generated into `/doc` using the command `make doc`.*
## AWFUL WARNING 1
This does not work. It isn't likely to work any time soon. If you want to learn Lisp, don't start here; try Clojure, Scheme or Common Lisp (in which case I recommend Steel Bank Common Lisp). If you want to learn how Lisp works, still don't start here. This isn't ever going to be anything like a conventional Lisp environment.
What it sets out to be is a Lisp-like system which:
* Can make use (albeit not, at least at first, very efficiently) of machines with at least [Zettabytes](http://highscalability.com/blog/2012/9/11/how-big-is-a-petabyte-exabyte-zettabyte-or-a-yottabyte.html) of RAM;
* Can make reasonable use of machines with at least tens of thousands of processors;
* Can concurrently support significant numbers of concurrent users, all doing different things, without them ever interfering with one another;
* Can ensure that users cannot escalate privilege;
* Can ensure users private data remains private.
When Linus Torvalds sat down in his bedroom to write Linux, he had something usable in only a few months. BUT:
* Linus was young, energetic, and extremely talented; I am none of those things.
* Linus was trying to build a clone of something which already existed and was known to work. Nothing like what I'm aiming for exists.
* Linus was able to adopt the GNU user space stack. There is no user space stack for this idea; I don't even know what one would look like.
## AWFUL WARNING 2
This project is necessarily experimental and exploratory. I write code, it reveals new problems, I think about them, and I mutate the design. This documentation does not always keep up with the developing source code.

8
docs/Homogeneity.md
View file

@ -1,8 +0,0 @@
# Homogeneity
A homogeneity is a [regularity](Regularity.html) which has a validation funtion associated with each key. A member can only be added to a homogeneity if not only does it have all the required keys, but the value of each key in the candidate member satisfies the validation function for that key. For example, the validation function for the age of a person might be something like
```
(fn [value]
(and (integer? value) (positive? value) (< value 140)))
```

2
docs/How-do-we-notate-paths.md
View file

@ -4,7 +4,7 @@ In order to make the namespaces thing work, we need a convenient way to notate p
In this discussion, a **namespace** is just a named, mutable hashmap (but not necessarily mutable by all users; indeed namespaces will almost always be mutable only by selected users. I cannot presently see a justified use for a publicly writable namespace). '**Named**', of a hashmap, merely means there is some path from the privileged root namespace which is the value of `oblist` which leads to that hashmap. A **path** is in principle just a sequence of keys, such that the value of each successive key is bound to a namespace in the namespace bound by its predecessor. The evaluable implementation of paths will be discussed later.
I think also that there must be a privileged **session** namespace, containing information about the current session, which the user can read but not write. The session namespace will be indicated by the privileged name '&sect;'.
I think also that there must be a privileged **session** namespace, containing information about the current session, which the user can read but not write.
## Security considerations

72
docs/Implementing-post-scarcity-hardware.md
View file

@ -1,72 +0,0 @@
The address space hinted at by using 64 bit cons-space and a 64 bit vector space containing objects each of whose length may be up to 1.4e20 bytes (2^64 of 64 bit words) is so large that a completely populated post-scarcity hardware machine can probably never be built. But that doesn't mean I'm wrong to specify such an address space: if we can make this architecture work for machines that can't (yet, anyway) be built, it will work for machines that can; and, changing the size of the pointers, which one might wish to do for storage economy, can be done with a few edits to consspaceobject.h.
But, for the moment, let's discuss a potential 32 bit psh machine, and how it might be built.
## Pass one: a literal implementation
Let's say a processing node comprises a two core 32 bit processor, such as an ARM, 4GB of RAM, and a custom router chip. On each node, core zero is the actual processing node, and core one handles communications. We arrange these on a printed circuit board that is 4 nodes by 4 nodes. Each node is connected to the nodes in front, behind, left and right by tracks on the board, and by pins to the nodes on the boards above and below. On the edges of the board, the tracks which have no 'next neighbour' lead to some sort of reasonably high speed bidirectional serial connection &mdash; I'm imagining optical fibre (or possibly pairs of optical fibre, one for each direction). These boards are assembled in stacks of four, and the 'up' pins on the top board and the 'down' pins (or sockets) on the bottom board connect to similar high speed serial connectors.
This unit of 4 boards &mdash; 64 compute nodes &mdash; now forms both a logical and a physical cube. Let's call this cube module a crystal. Connect left to right, top to bottom and back to front, and you have a hypercube. But take another identical crystal, place it along side, connect the right of crystal A to the left of crystal B and the right of B to the left of A, leaving the tops and bottoms and fronts and backs of those crystals still connected to themselves, and you have a larger cuboid with more compute power and address space but slightly lower path efficiency. Continue in this manner until you have four layers of four crystals, and you have a compute unit of 4096 nodes. So the basic 4x4x4 building block &mdash; the 'crystal' &mdash; is a good place to start, and it is in some measure affordable to build &mdash; low numbers of thousands of pounds, even for a prototype.
I imagine you could get away with a two layer board &mdash; you might need more, I'm no expert in these things, but the data tracks between nodes can all go on one layer, and then you can have a raster bus on the other layer which carries power, backup data, and common signals (if needed).
So, each node has 4Gb of memory (or more, or less &mdash; 4Gb here is just illustrative). How is that memory organised? It could be treated as a heap, or it could be treated as four separate pages, but it must store four logical blocks of data: its own curated conspage, from which other nodes can request copies of objects; its own private housekeeping data (which can also be a conspage, but from which other nodes can't request copies); its cache of copies of data copied from other nodes; and its heap.
Note that a crystal of 64 nodes each with 4Gb or RAM has a total memory of 256Gb, which easily fits onto a single current generation hard disk or SSD module. So I'm envisaging that either the nodes take turns to back up their memory to backing store all the time during normal operation. They (obviously) don't need to backup their cache, since they don't curate it.
What does this cost? About £15 per processor chip, plus £30 for memory, plus the router, which is custom but probably still in tens of pounds, plus a share of the cost of the board; probably under £100 per node, or £6500 for the 'crystal'.
## Pass two: a virtual implementation
OK, OK, this crystal cube is a pretty concept, but let's get real. Using one core of each of 64 chips makes the architecture very concrete, but it's not necessarily efficient, either computationally or financially.
64 core ARM chips already exist:
1. [Qualcom Hydra](https://eltechs.com/hydra-is-the-name-of-qualcomms-64-core-arm-server-processor/) - 64 of 64 bit cores;
2. [Macom X-Gene](https://www.apm.com/products/data-center/x-gene-family/x-gene/) - 64 of 64 bit cores;
2. [Phytium Mars](https://www.nextplatform.com/2016/09/01/details-emerge-chinas-64-core-arm-chip/) - 64 cores, but frustratingly this does not say whether cores are 32 or 64 bit
There are other interesting chips which aren't strictly 64 core:
1. [Cavium ThunderX](https://www.servethehome.com/exclusive-first-cavium-thunderx-dual-48-core-96-core-total-arm-benchmarks) - ARM; 96 cores, each 64 bit, in pairs of two, shipping now;
2. [Sparc M8](https://www.servethehome.com/oracle-sparc-m8-released-32-cores-256-threads-5-0ghz/) - 32 of 64 bit cores each capable of 8 concurrent threads; shipping now.
## Implementing the virtual hypercube
Of course, these chips are not designed as hypercubes. We can't route our own network of physical connections into the chips, so our communications channels have to be virtual. But we can implement a communications channel as a pair of buffers, an 'upstream' buffer writable by the lower-numbered processor and readable by the higher, and a 'downstream' buffer writable by the higher numbered processor and readable by the lower. Each buffer should be at least big enough to write a whole cons page object into, optionally including a cryptographic signature if that is implemented. Each pair of buffers also needs at least four bits of flags, in order to be able, for each direction, to be able to signal
0. Idle &mdash; the processor at the receiving end is idle and can accept work;
1. Busy writing &mdash; the processor at the sending end is writing data to the buffer, which is not yet complete;
2. Ready to read &mdash; the processor at the sending end has written data to the buffer, and it is complete;
3. Read &mdash; the processor at the receiving end has read the current contents of the buffer.
Thus I think it takes at least six clock ticks to write the buffer (set busy-writing, copy four 64 bit words into the buffer, set ready-to-read) and five to read it out &mdash; again, more if the messages are cryptographically signed &mdash; for an eleven clock tick transfer (the buffers may be allocated in main memory, but in practice they will always live in L2 cache). That's probably cheaper than making a stack frame. All communications channels within the 'crystal' cost exactly the same.
But note! As in the virtual design, a single thread cannot at the same time execute user program and listen to communications from neighbours. So a node has to be able to run two threads. Whether that's two threads on a single core, or two cores per node, is a detail. But it makes the ThunderX and Spark M8 designs look particularly interesting.
But note that there's one huge advantage that this single-chip virtual crystal has over the literal design: all cores access the same memory pool. Consequently, vector space objects never have to be passed hop, hop, hop across the communications network, all can be accessed directly; and to pass a list, all you have to pass is its first cons cell. So any S-Expression can be passed from any node to any of its 6 proximal neighbours in one hop.
There are downsides to this, too. While communication inside the crystal is easier and quicker, communication between crystals becomes a lot more complex and I don't yet even have an idea how it might work. Also, contention on the main address bus, with 64 processors all trying to write to and read from the same memory at the same time, is likely to be horrendous, leading to much lower speed than the solution where each node has its own memory.
On a cost side, you probably fit this all onto one printed circuit board as against the 4 of the 'literal' design; the single processor chip is likely to cost around £400; and the memory will probably be a little cheaper than on the literal design; and you don't need the custom routers, or the connection hardware, or the optical transceivers. So the cost probably looks more like £5,000. Note also that this virtual crystal has 64 bit processors (although address bus contention will almost certainly burn all that advantage and more).
An experimental post-scarcity machine can be built now &mdash; and I can almost afford to build it. I don't have the skills, of course; but I can learn.
## Size of a fully populated machine
### Memory size
To fully implement the software specification as currently written, each node would need 128Gb of RAM for its curated cons space alone (since we can have 2<sup>32</sup> cons cells each of 32 bytes); an amount of memory for vector space; substantial cache of objects being processed by the node but curated by other nodes; and scratchpad space.
How much memory for vector space? The current software specification allows for vectors up to 32 times the total address space of currently available 64 bit processors. But not only could such objects not easily be stored with current generation technology, they could also not be copied across the hypercube lattice in any useful sort of time. So functions which operate on large vector space objects would necessarily have to migrate to the node where the object is curated, rather than have the object migrate. I don't currently have an account of how this could be done.
However, obviously it is unaffordable to build a machine which can explore problems like that as a first prototype, so this is at present academic.
### Lattice size
If we hold to the doctrine of one cons page per node, which has the advantage of making addressing reasonably simple, then there can be up to 2<sup>32</sup>, or 4,294,967,296 nodes, forming a hypercube of 1625 x 1625 x 1625 nodes. The total address space of this machine would be of the order of 79,228,162,514,264,337,593,543,950,336 bytes, or 7.9x10<sup>28</sup>. This is about 7 brontobytes - far beyond the zetabytes of my original sketch.
Hello, I seem to have designed a computer which would terrify even the [Magratheans](https://hitchhikers.fandom.com/wiki/Magrathea).

98
docs/Interning-strings.md
View file

@ -1,98 +0,0 @@
# Interning strings
I'm trying to understand what it means to intern a name in an environment with a messy and possibly shifting graph of namespaces.
My current thinking is that in data terms a name is just a string. In evaluation terms, an unquoted string (in lexical terms one unprotected by enclosing quotation marks) is a name, while a quoted string is 'a string'. So, supposing the name **froboz** is not bound in the current environment,
(eval froboz)
causes an unbound variable exception to be thrown, while
(eval "froboz")
returns the value **"froboz"**. This begs the question of whether there's any difference between **"froboz"** and **'froboz**, and the answer is that at this point I don't know.
There will be a concept of a root [namespace](Namespace.html), in which other namespaces may be bound recursively to form a directed graph. Because at least some namespaces are mutable, the graph is not necessarily acyclic. There will be a concept of a current namespace, that is to say the namespace in which the user is currently working.
There must be some notation to say distinguish a request for the value of a name in the root namespace and the value of a name in the current namespace. For now I'm proposing that:
(eval froboz)
will return the value that **froboz** is bound to in the current namespace;
(eval .froboz)
will return the value that **froboz** is bound to in the root namespace;
(eval foobar.froboz)
will return the value that **froboz** is bound to in a namespace which is the value of the name **foobar** in the current namespace; and that
(eval .system.users.simon.environment.froboz)
will return the value that **froboz** is bound to in the environment of the user of the system called **simon**.
The exact path separator syntax may change, but the principal that when interning a symbol it is broken down into a path of tokens, and that the value of each token is sought in a namespace bound to the previous token, is likely to remain.
Obviously if **froboz** is interned in one namespace it is not necessarily interned in another, and vice versa. There's a potentially nasty problem here that two lexically identical strings might be bound in different namespaces, so that there is not one canonical interned **froboz**; if this turns out to cause problems in practice there will need to be a separate canonical [hashtable](Hashtable.html) of individual path elements.
Obviously this means there may be arbitrarily many paths which reference the same data item. This is intended.
## Related functions
### (intern! string)
Binds *string*, considered as a path, to **NIL**. If some namespace along the path doesn't exist, throws an exception. Obviously if the current user is not entitled to write to the terminal namespace, also throws an exception.
### (intern! string T)
Binds *string*, considered as a path, to **NIL**. If some namespace along the path doesn't exist, create it as the current user with both read and write [access control](Access-control.html) lists taken from the current binding of **friends** in the current environment. Obviously if the current user is not entitled to write to the last pre-existing namespace, throws an exception.
### (intern! string T write-access-list)
Binds *string*, considered as a path, to **NIL**. If some namespace along the path doesn't exist, create it as the current user with the read [access control](https://www.journeyman.cc/blog/posts-output/2006-02-20-postscarcity-software/) list taken from the current binding of **friends** in the current environment, and the write access control list taken from the value of *write-access-list*. Obviously if the current user is not entitled to write to the last pre-existing namespace, throws an exception.
### (set! string value)
Binds *string*, considered as a path, to *value*. If some namespace along the path doesn't exist, throws an exception. Obviously if the current user is not entitled to write to the terminal namespace, also throws an exception.
### (set! string value T)
Binds *string*, considered as a path, to *value*. If some namespace along the path doesn't exist, create it as the current user with both read and write [access control](Access-control.html) lists taken from the current binding of **friends** in the current environment. Obviously if the current user is not entitled to write to the last pre-existing namespace, throws an exception.
### (set! string value T write-access-list)
Binds *string*, considered as a path, to *value*. If some namespace along the path doesn't exist, create it as the current user with the read [access control](Access-control.html) list taken from the current binding of **friends** in the current environment, and the write access control list taken from the value of *write-access-list*. Obviously if the current user is not entitled to write to the last pre-existing namespace, throws an exception.
### (put! string token value)
Considers *string* as the path to some namespace, and binds *token* in that namespace to *value*. *Token* should not contain any path separator syntax. If the namespace doesn't exist or if the current user is not entitled to write to the namespace, throws an exception.
### (string-to-path string)
Behaviour as follows:
(string-to-path "foo.bar.ban") => ("foo" "bar" "ban")
(string-to-path ".foo.bar.ban") => ("" "foo" "bar" "ban")
Obviously if the current user can't read the string, throws an exception.
### (path-to-string list-of-strings)
Behaviour as follows:
(path-to-string '("foo" "bar" "ban")) => "foo.bar.ban"
(path-to-string '("" "foo" "bar" "ban")) => ".foo.bar.ban"
Obviously if the current user can't read some element of *list-of-strings*, throws an exception.
### (interned? string)
Returns a string lexically identical to *string* if *string*, considered as a path, is bound (i.e.
1. all the non-terminal elements of the path are bound to namespaces,
2. all these namespaces are readable by the current user;
3. the terminal element is bound in the last of these;
Returns nil if *string* is not so bound, but all namespaces along the path are readable by the current user.
I'm not certain what the behaviour should be if some namespace on the path is not readable. The obvious thing is to throw an access control exception, but there are two possible reasons why not:
1. It may turn out that this occurs too often, and this just becomes a nuisance;
2. From a security point of view, is evidence that there is something there to which you don't have access (but that is in a sense an argument against ever throwing an access exception at all).

36
docs/Lazy-Collections.md
View file

@ -1,36 +0,0 @@
# Lazy Collections
If we're serious about a massively parallel architecture, and especially if we're serious about passing argument evaluation off to peer processors, then it would be madness not to implement lazy collections.
Architecturally lazy collections violate the 'everything is immutable' rule, but only in a special case: you can do the equivalent of `replacd` on a cell that forms part of lazy collection, but only as you create it, before you release it to another process to be consumed; and you may only do it once, to replace the `NIL` list termination value with a pointer to a new lazy sell.
From the consumer's point of view, assuming for the moment that the consumer is a peer processor, it looks just like a list, except that when you request an element which is, as it were, at the write cursor &mdash; in other words, the producing process hasn't yet generated the next element in the collection &mdash; the request will block.
## How are lazy sequences created?
Essentially, lazy collections are created by a very few primitive functions; indeed, it may boil down to just two,`read` which reads characters from a stream, and `mapc` which applies a function to successive values from a sequence. `reduce` may be a third lazy-generating function, but I suspect that it may be possible to write `reduce` using `mapc`.
## What does a lazy sequence look like?
Essentially you can have a lazy sequence of objects, which looks exactly like a list except that it's lazy, of a lazy sequence of characters, which looks exactly like a string except that it's lazy. For practical purposes it would be possible for `mapc` to generate perfectly normal `CONS` cells and for `read` to generate perfectly normal `STRG` cells, but we've actually no shortage of tags, and I think it would be useful for debugging and also for process scheduling to know whether one is dealing with a lazy construct or not. so I propose three new tags:
* `LZYC` &mdash; like a cons cell, but lazy;
* `LZYS` &mdash; like a string cell, but lazy;
* `LZYW` &mdash; the thing at the end of a lazy sequence which does some work when kicked.
I acknowledge that, given that keywords and symbols are also sequences of characters, one might also have lazy symbols and lazy keywords but I'm struggling to think of situations in which these would be useful.
## How do we compute with lazy sequences in practice?
Consider the note [parallelism](Parallelism.html). Briefly, this proposes that a compile time judgement is made at the probable cost of evaluating each argument; that the one deemed most expensive to evaluate is reserved to be evaluated on the local node, and for the rest, a judgement is made as to whether it would be cheaper to hand them off to peers or to evaluate them locally. Well, for functions which return lazies –– and the compiler should certainly be able to infer whether a function will return a lazy &mdash; it will always make sense to hand them off, if there is an available idle peer to which to hand off. In fact, lazy-producers are probably the most beneficial class of function calls to hand off, since, if handed off to a peer, the output of the function can be consumed without any fancy scheduling on the local node. Indeed, if all lazy-producers can be reliably handed off, we probably don't need a scheduler at all.
## How do lazy sequences actually work?
As you iterate down a lazy list, you may come upon a cell whose `CDR` points to a 'Lazy Worker' (`WRKR`) cell. This is a pointer to a function. You (by which I mean the CDR function &mdash; just the ordinary everyday CDR function) mark the cell as locked, and call the function.
1. It may compute and return the next value; or
2. it may return a special marker (which cannot be `NIL`, since that's a legitimate value), indicating that there will be no further values; or
3. it may block, waiting for the next value to arrive from a stream or something; or
4. it may return a special value (which I suspect may just be its own address) to indicate that 'yes, there are in principle more values to come but they're not ready yet'.
In cases 1 above, you replace the `CDR` (hard replace &mdash; actual immutability defying change) of the sequence cell you were looking at with a new sequence cell of the same type whose `CAR` points to the newly delivered value and whose `CDR` points to the `WRKR` cell.

80
docs/Memory-management.md
View file

@ -1,80 +0,0 @@
# Memory management
Most of the memory management ideas that I want to experiment with this thing are documented in essays on my blog:
1. [Functional languages, memory management, and modern language runtimes](https://www.journeyman.cc/blog/posts-output/2013-08-23-functional-languages-memory-management-and-modern-language-runtimes/)
2. [Reference counting, and the garbage collection of equal sized objects](https://www.journeyman.cc/blog/posts-output/2013-08-25-reference-counting-and-the-garbage-collection-of-equal-sized-objects/)
Brief summary:
# The problem
My early experience included Lisps with older, pre-generational, mark and sweep garbage collectors. The performance of these was nothing like as bad as some modern texts claim, but they did totally halt execution of the program for a period of seconds at unpredictable intervals. This is really undesirable.
I became interested in reference counting garbage collectors, because it seemed likely that these would generate shorter pauses in execution. But received wisdom was that mark-and-sweep outperformed reference counting (which it probably does, overall), and that in any case generational garbage collection has so improved mark and sweep performance that the problem has gone away. I don't wholly accept this.
## Separating cons space from vector space
Lisps generate lots and lots of very small, equal sized objects: cons cells and other things which are either the same size as or even smaller than cons cells and which fit into the same memory footprint. Furthermore, most of the volatility is in cons cells &mdash; they are often extremely short lived. Larger objects are allocated much more infrequently and tend to live considerably longer.
Because cons cells are all the same size, and because integers and doubles fit into the memory footprint of a cons cell, if we maintain an array of memory units of this size then we can allocate them very efficiently because we never have to move them &mdash; we can always allocate a new object in memory vacated by deallocating an old one. Deallocation is simply a matter of pushing the deallocated cell onto the front of the free list; allocation is simply a matter of popping a cell off the free list.
By contrast, a conventional software heap fragments exactly because we allocate variable sized objects into it. When an object is deallocated, it leaves a hole in the heap, into which we can only allocate objects of the same size or smaller. And because objects are heterogeneously sized, it's probable that the next object we get to allocate in it will be smaller, leaving even smaller unused holes.
Consequently we end up with a memory like a swiss cheese &mdash; by no means fully occupied, but with holes which are too small to fit anything useful in. In order to make memory in this state useful, you have to mark and sweep it.
So my first observation is that [cons space](Cons-space.html) and what I call [vector space](Vector-space.html) &mdash; that is, the heap into which objects which won't fit into the memory footprint of a cons cell are allocated &mdash; are systematically different and require different garbage collection strategies.
## Reference counting: the objections
### Lockin from reference count overflow
Older reference counting Lisps tended to allocate very few bits for the reference count. Typically a cons cell was allocated in 32 bits, with two twelve bit pointers, leaving a total of eight bits for the tag bits and the reference counter bits. So you had reference counters with only eight or sixteen possible values. When a reference counter hits the top value which can be stored in its field, it cannot be further incremented. So you cannot ever decrement it, because you don't know whether it represents `max-value` or, e.g., `max-value + 21`. So any cell which ended up with that number of other cells pointing to it, even if only very temporarily, got locked into memory and couldn't be garbage collected.
But modern computers have vastly more memory than the computers on which those Lisps ran. My desktop machine now has more than 16,000 times as much memory as my very expensive workstation of only thirty years ago. We can afford much bigger reference counter fields. So the risk of hitting the maximum reference count value is much lower.
### Lockin from circular data structures
The other 'fault' of older reference counting Lisps is that in older Lisps, cons cells were not immutable. There were functions *RPLACA* and *RPLACD* which overwrote the value of the CAR and CDR pointers of a cons cell respectively. Thus it was possible to create circular data structures. In a reference counting Lisp, a circular data structure can never be garbage collected even if nothing outside the circle any longer points to it, because each cell in the circle is pointed to by another cell in the circle. Worse, any data structure outside the circle that is pointed to by a cell in the circle also cannot ever be garbage collected.
So badly designed programs on reference counting Lisps could leak memory badly and consequently silt up and run out of allocatable store.
But modern Lisps &mdash; like Clojure &mdash; use immutable data structures. The nature of immutable data structures is that an older node can never point to a newer node. So circular data structures cannot be constructed.
### Performance
When a memory management system with a reference counting garbage collector allocates a new cons cell, it needs to increment the reference counts on each of the cells the new cell points to. One with a mark-and-sweep garbage collector doesn't have to do this. When the system deallocates a cell, it has to decrement the counts on each of the cells it pointed to. One with a mark-and-sweep garbage collector doesn't have to do this. Overall, many well informed people have claimed to me, the performance of memory management system with a mark-and-sweep collector is a bit better than one with a reference counting garbage collector.
That's probably true.
## Why I think reference counting is nevertheless a good idea
A reference counting garbage collector does a little bit of work every time a cell (or something else that points to a cell) is deallocated. It's unlikely ever to pause the system for noticeable time. In the extreme case, when you remove the one link that points to a massive, complex data structure, the cascade of deallocations might produce a noticeable pause but in most programs that's a very rare occurrence. You could even run the deallocator in a separate thread, evening out its impact on performance even further.
Also, over the thirty years, the performance of our computers has also improved immeasurably. My desktop machine now has over 6,000 times the performance of a DEC VAX. We can afford to be a little bit inefficient. If reference counting were to prove to be 10% slower overall, it might still be preferable if it gave smoother performance.
## Separate freelists for different sized holes
We're still left with the possibility of fragmenting heap space in the manner I've described. Doing clever memory management of heap space can't be done if we depend on the memory management functions provided by the C or Rust compilers and their standard libraries, because while we know the size and location of memory allocations they've done for Lisp objects, we don't necessarily know the size or location of memory allocations they've made for other things.
So if we're going to do clever memory management of heap space, we're probably going to have to claim heap space from the system in pages and allocate our objects within these pages. This bothers me both because there's a possibility of space wasted at the edges of the pages, and because it becomes more complex to allocate objects that are bigger than a page.
The alternative is to bypass the standard library memory allocation and just allocate it ourselves, which will be tricky; but we can then manage the whole of the address space available to our process.
Either way, most of the Lisp objects we allocate will be of a few sizes. A default hashtable is always yay big, for example; it has this many buckets. How many 'this many' is I don't yet know, but the point is it will be a standard number. When a hashtable overflows that standard size, we'll automatically replace it with another of a bigger, but still standard size.
We're going to store strings in cons space, not vector space. There aren't a lot of things we're going to store in vector space. Probably raster images, or sound objects, will be the only ones which are truly variable in size.
So if we have three or four fixed sizes which accommodate most of our vector space objects, we should create free lists for each of those sizes. When deallocating a default sized hashtable, for example, we'll link the newly freed memory block onto a special purpose freelist of blocks the size of a default sized hashtable; when we want to allocate a new default sized hashtable, we'll first check that freelist and, if it has one, pop it from there, before we ask the system to allocate new virgin memory.
## Conclusion
Even the 'separate freelists for different sized objects' idea won't prevent vector space fragmenting, although it may slow it considerably. Ultimately, mark and sweep of vector space will be needed and will be very expensive. It can be made much less expensive by dividing cons pointers into a page number field and an offset field rather than being a simple memory address, but that will make all memory access slower, so it too is a trade off.
There are several possible approaches to optimising garbage collection. I want to try as many as possible of these approaches, although I'm not convinced that any of them will prove in the end better than a conventional generational garbage collector. Still, they're worth trying.
What we need to do is build a system into which different memory management subsystems can be plugged, and to develop a set of benchmarks which aggressively test memory allocation and deallocation, and then see how the different possibilities perform in practice.
## Further reading
I've written a [number of essays](https://blog.journeyman.cc/search/label/Memory%20management) on memory management, of which I'd particularly point you to [Reference counting, and the garbage collection of equal sized objects](https://blog.journeyman.cc/2013/08/reference-counting-and-garbage.html).

10
docs/Names-of-things.md
View file

@ -1,10 +0,0 @@
* **assoc list** An assoc list is any list all of whose elements are cons-cells.
* **association** Anything which associates names with values. An *assoc list* is an association, but so it a *map*, a *namespace*, a *regularity* and a *homogeneity*.
* **homogeneity** A [homogeneity](Homogeneity.html) is a *regularity* which has a validation funtion associated with each key.
* **keyword** A [keyword](Keyword.html) is a token whose denotation starts with a colon and which has a limited range of allowed characters, not including punctuation or spaces, which evaluates to itself irrespective of the current binding environment.
* **map** A map in the sense of a Clojure map; immutable, adding a name/value results in a new map being created. A map may be treated as a function on *keywords*, exactly as in Clojure.
* **namespace** A namespace is a mutable map. Generally, if a namespace is shared, there will be a path from the oblist to that namespace.
* **oblist** The oblist is a privileged namespace which forms the root of all canonical paths. It is accessed at present by the function `(oblist)`, but it can be denoted in paths by the empty keyword.
* **path** A [path](How-do-we-notate-paths.html) is a list of keywords, with special notation and semantics.
* **regularity** A [regularity](Regularity.html) is a map whose values are maps, all of whose members share the same keys. A map may be added to a regularity only if it has all the keys the regularity expects, although it may optionally have more. It is legitimate for the same map to be a member of two different regularities, if it has a union of their keys. Keys in a regularity must be keywords. Regularities are roughly the same sort of thing as objects in object oriented programming or tables in databases, but the values of the keys are not policed (see `homogeneity`).

43
docs/Parallelism.md
View file

@ -1,43 +0,0 @@
# Parallelism
If this system doesn't make reasonably efficient use of massively parallel processors, it's failed. The sketch hardware for which it's designed is [Post Scarcity Hardware](Post-scarcity-hardware.html); that system probably won't ever exist but systems somewhat like it almost certainly will, because we're up against the physical limits on the performance of a von Neumann machine, and the only way we can increase performance now is by going increasingly parallel.
So on such a system, every function invocation may normally delegate every argument to a different processor, if there is another processor free (which there normally will be). Only special forms, like *cond*, which implement explicit flow control, should serialise evaluation.
Therefore the semantics of every function must assume that the order of the evaluation of arguments is undetermined, and that the environment in which each argument is evaluated cannot be influenced by the evaluation of any other argument. Where the semantics of a function violate this expectation, this must be made explicit in the documentation and there should probably be a naming convention indicating this also.
This means, for example, that `and` and `or` cannot be used for flow of control.
## Determining when to hand off computation of an argument to another node
There's obviously a cost of transferring an argument to another node. The argument must be passed to the peer, and, if the argument points to something which isn't already in the peer's cache, the peer will need to fetch that something; and finally, the peer will have to pass the result back.
Communications links to other nodes will be of varying speeds. Communication with a peer which is on the same piece of silicon will be fast, with a peer accessed over a parallel slower, with a peer accessed over a serial bus slower still. So one of the things each node must do in boot is to handshake with each of its proximal neighbours and determine the connection speed. This gives us a feeling for the cost of handoff.
The other variable is the cost of computation.
Suppose we are evaluating
```
(add 2 2)
```
Then there's clearly no point in handing anything off to a peer, since the arguments each evaluate to themselves.
If we're evaluating
```
(add (cube-root pi) (multiply 2 2) (factorial 1000))
```
Then clearly the first and third arguments are going to be costly to compute. As a programmer, I can see that by inspection; the goal has to be for the compiler to be able to assign functions to cost classes.
If the node which starts the evaluation evaluates the most costly argument itself, then it's reasonable to suppose that by the time it's finished that the neighbours to whom it handed off the other two will have completed and returned their results (or will do so sooner than the original node could compute them). So the heuristics seem to be
1. Don't hand off anything which evaluates to itself;
2. Don't hand off the argument the compiler predicts will be most expensive;
3. Only hand off to a slow to reach neighbour arguments the compiler predicts will be very expensive.
Note that there will be different costs here depending whether we have one memory map per node (i.e. each node curates one cons page and its own vector space, and must copy objects from other nodes into its own space before it can compute on them), or one memory map per crystal, in which case nodes on the same crystal do not need to copy data across.
Copying data, especially if it comes from a node further than a proximal neighbour, will be extremely expensive.

9
docs/Paths.md
View file

@ -1,9 +0,0 @@
# Paths
*See also [How do we notate paths?](How do we notate paths?.html), which in part supercedes this.*
A path is essentially a list of keywords.
However, the path `(:aardvark :beluga :coeleocanth :dodo)` may be denoted `:aardvark:beluga:coeleocanth:dodo` and will be expanded into a list by the reader. If the value of `:ardvark` in the current environment is a map, and the value of `:beluga` in that map is a map, and the value of `:coeleocanth` in that map is a map, then the value of the path `:aardvark:beluga:coeleocanth:dodo` is whatever the value of `:dodo` is in the map indicated by `:aardvark:beluga:coeleocanth`; however if the path cannot be fully satisfied the value is `nil`.
The notation `::aardvark:beluga:coeleocanth:dodo` is expanded by the reader into `((oblist) :aardvark :beluga :coeleocanth :dodo)`, or, in other words, is a path from the root namespace.

12
docs/Plan-overview.md
View file

@ -1,12 +0,0 @@
# Plan overview
1. Specify enough of memory arrangement and core functions that I can build something that can start, read (parse) a stream, allocate some memory, print what it's got, and exit. *(achieved)*
2. Build the above. *(achieved)*
3. Fully specify eval/apply, lambda, and def; make a decision whether to eval compiled code, interpreted code, or (at this stage) both. In the long term I want to eval compiled code only, but that requires working out how to generate the right code! *(achieved)*
4. Build enough of the system that I can write and evaluate new functions. *(achieved)*
5. Build enough of the system that two users can log in simultaneously with different identities, so as to be able to test user separation and privacy, but also sharing.
6. Try seriously to get compilation working.
7. Get system persistence working.
8. Get an emulated multi-node system working.
9. Get a bare-metal node working.
10. Get a hypercube (even if of only 2x2x2 nodes) working.

180
docs/Post-scarcity-hardware.md
View file

@ -1,180 +0,0 @@
# Implementing post scarcity hardware
_I wrote this essay in 2014; it was previously published on my blog, [here](https://www.journeyman.cc/blog/posts-output/2017-09-19-implementing-postscarcity-hardware/)_
Eight years ago, I wrote an essay which I called [Post Scarcity Software](Post-scarcity-software.html). It's a good essay; there's a little I'd change about it now - I'd talk more about the benefits of immutability - but on the whole it's the nearest thing to a technical manifesto I have. I've been thinking about it a lot the last few weeks. The axiom on which that essay stands is that modern computers - modern hardware - are tremendously more advanced than modern software systems, and would support much better software systems than we yet seem to have the ambition to create.
That's still true, of course. In fact it's more true now than it was then, because although the pace of hardware change is slowing, the pace of software change is still glacial. So nothing I'm thinking of in terms of post-scarcity computing actually needs new hardware.
Furthermore, I'm a software geek. I know very little about hardware; but I'm very much aware that as parallelism increases, the problems of topology in hardware design get more and more difficult. I've no idea how physically to design the machines I'm thinking of. But nevertheless I have been thinking more and more, recently, about the design of post-scarcity hardware to support post-scarcity software.
And I've been thinking, particularly, about one issue: process spawning on a new processor, on modern hardware, with modern operating systems, is ridiculously expensive.
# A map of the problem
What got me thinking about this was watching the behaviour of the [Clojure](http://clojure.org/) map function on my eight core desktop machine.
Mapping, in a language with immutable data, in inherently parallelisable. There is no possibility of side effects, so there is no particular reason for the computations to be run serially on the same processor. MicroWorld, being a cellular automaton, inherently involves repeatedly mapping a function across a two dimensional array. I was naively pleased that this could take advantage of my modern hardware - I thought - in a way in which similarly simple programs written in Java couldn't...
...and then was startled to find it didn't. When running, the automaton would camp on a single core, leaving the other seven happily twiddling their thumbs and doing minor Unixy background stuff.
What?
It turns out that Clojure's default *map* function simply serialises iterations in a single process. Why? Well, one finds out when one investigates a bit. Clojure provides two different versions of parallel mapping functions, *pmap* and *clojure.core.reducers/map*. So what happens when you swap *map* for *pmap*? Why, performance improves, and all your available cores get used!
Except...
Performance doesn't actually improve very much. Consider this function, which is the core function of the [MicroWorld](https://www.journeyman.cc/blog/posts-output/2014-08-26-modelling-settlement-with-a-cellular-automaton/) engine:
<pre>
(defn map-world
"Apply this `function` to each cell in this `world` to produce a new world.
the arguments to the function will be the world, the cell, and any
` additional-args` supplied. Note that we parallel map over rows but
just map over cells within a row. That's because it isn't worth starting
a new thread for each cell, but there may be efficiency gains in
running rows in parallel."
([world function]
(map-world world function nil))
([world function additional-args]
(into []
(pmap (fn [row]
(into [] (map
#(apply function
(cons world (cons % additional-args)))
row)))
world))))
</pre>
As you see, this maps across a two dimensional array, mapping over each of the rows of the array, and, within each row, mapping over each cell in the row. As you can see, in this current version, I parallel map over the rows but serial map over the cells within a row.
Here's why:
## Hybrid parallel/non-parallel version
This is the current default version. It runs at about 650% processor loading - i.e. it maxes out six cores and does some work on a seventh. The eighth core is doing all the Unix housekeeping.
(time (def x1 (utils/map-world
(utils/map-world w heightmap/tag-altitude (list hm))
heightmap/tag-gradient)))
"Elapsed time: 24592.327364 msecs"
#'mw-explore.optimise/x1
## Pure parallel version
Runs at about 690% processor loading - almost fully using seven cores. But, as you can see, fully one third slower.
(time (def x2 (utils/map-world-p-p
(utils/map-world-p-p w heightmap/tag-altitude (list hm))
heightmap/tag-gradient)))
"Elapsed time: 36762.382725 msecs"
#'mw-explore.optimise/x2
(For completeness, the *clojure.core.reducers/map* is even slower, so is not discussed in any further detail)
## Non parallel version
Maxes out one single core, takes about 3.6 times as long as the hybrid version. But, in terms of processor cycles, that's a considerable win - because 6.5 cores for 24 seconds is 156 seconds, so there's a 73% overhead in running threads across multiple cores.
(time (def x2 (utils/map-world-n-n
(utils/map-world-n-n w heightmap/tag-altitude (list hm))
heightmap/tag-gradient)))
"Elapsed time: 88412.883849 msecs"
#'mw-explore.optimise/x2
Now, I need to say a little more about this. It's obvious that there's a considerable set-up/tear-down cost for threads. The reason I'm using *pmap* for the outer mapping but serial *map* for the inner mapping rather than the other way round is to do more work in each thread.
However, I'm still simple-mindedly parallelising the whole of one map operation and serialising the whole of the other. This particular array is 2048 cells square - so over four million cells in total. But, by parallelising the outer map operation, I'm actually asking the operating system for 2048 threads - far more than there are cores. I have tried to write a version of map using [Runtime.getRuntime().availableProcessors()](http://stackoverflow.com/questions/1980832/java-how-to-scale-threads-according-to-cpu-cores) to find the number of processors I have available, and then partitioned the outer array into that number of partitions and ran the parallel map function over that partitioning:
(defn adaptive-map
"An implementation of `map` which takes note of the number of available cores."
[fn list]
(let [cores (.availableProcessors (. Runtime getRuntime ))
parts (partition-all (/ (count list) cores) list)]
(apply concat (pmap #(map fn %) parts))))
Sadly, as [A A Milne wrote](http://licoricelaces.livejournal.com/234435.html), 'It's a good sort of brake But it hasn't worked yet.'
But that's not what I came to talk about. I came to talk about the draft...
We are reaching the physical limits of the speed of switching a single processor. That's why our processors now have multiple cores. And they're soon going to have many more cores. Both Oracle ([SPARC](http://www.theregister.co.uk/2014/08/18/oracle_reveals_32core_10_beeellion_transistor_sparc_m7/)) and [ARM](http://www.enterprisetech.com/2014/05/08/arm-server-chips-scale-32-cores-beyond/) are demoing chips with 32 cores, each 64 bits wide, on a single die. [Intel and MIPS are talking about 48 core, 64 bit wide, chips](http://www.cpushack.com/2012/11/18/48-cores-and-beyond-why-more-cores/). A company called [Adapteva is shipping a 64 core by 64 bit chip](http://www.adapteva.com/products/silicon-devices/e64g401/), although I don't know what instruction set family it belongs to. Very soon we will have more; and, even if we don't have more cores on a physical die, we will have motherboards with multiple dies, scaling up the number of processors even further.
# The Challenge
The challenge for software designers - and, specifically, for runtime designers - is to write software which can use these chips reasonably efficiently. But the challenge, it seems to me, for hardware designers, is to design hardware which makes it easy to write software which can use it efficiently.
## Looking for the future in the past, part one
Thinking about this, I have been thinking about the [Connection Machine](http://en.wikipedia.org/wiki/Connection_Machine). I've never really used a Connection Machine, but there was once one in a lab which also contained a Xerox Dandelion I was working on, so I know a little bit about them. A Connection Machine was a massively parallel computer having a very large number - up to 65,536 - of very simple processors (each processor had a register width of one bit). Each processor node had a single LED lamp; when in use, actively computing something, this lamp would be illuminated. So you could see visually how efficient your program was at exploiting the computing resource available.
\[Incidentally while reading up on the Connection Machine I came across this [delightful essay](http://longnow.org/essays/richard-feynman-connection-machine/) on Richard Feynman's involvement in the project - it's of no relevance to my argument here, but nevertheless I commend it to you\]
The machine was programmed in a pure-functional variant of Common Lisp. Unfortunately, I don't know the details of how this worked. As I understand it each processor had its own local memory but there was also a pool of other memory known as 'main RAM'; I'm guessing that each processor's memory was preloaded with a memory image of the complete program to run, so that every processor had local access to all functions; but I don't know this to be true. I don't know how access to main memory was managed, and in particular how contention on access to main memory was managed.
What I do know from reading is that each processor was connected to twenty other processors in a fixed topology known as a hypercube. What I remember from my own observation was that a computation would start with just one or a small number of nodes lit, and flash across the machine as deeply recursive functions exploded from node to node. What I surmise from what I saw is that passing a computation to an unoccupied adjacent node was extremely cheap.
A possibly related machine from the same period which may also be worth studying but about which I know less was the [Meiko Computing Surface](http://www.new-npac.org/projects/cdroms/cewes-1999-06-vol1/nhse/hpccsurvey/orgs/meiko/meiko.html). The Computing Surface was based on the [Transputer T4](http://en.wikipedia.org/wiki/Transputer#T4:_32-bit) processor, a 32 bit processor designed specifically for parallel processing. Each transputer node had its own local store, and very high speed serial links to its four nearest neighbours. As far as I know there was no shared store. The Computing Surface was designed to be programmed in a special purpose language, [Occam](http://en.wikipedia.org/wiki/Occam_(programming_language)). Although I know that Edinburgh University had at one time a Computing Surface with a significant number of nodes, I don't know how many 'a significant number' is. It may have been hundreds of nodes but I'm fairly sure it wasn't thousands. However, each node was of course significantly more powerful than the Connection Machine's one bit nodes.
## A caveat
One of the lessons we learned in those high, far off, arrogant days was that special purpose hardware that could do marvellous things but was expensive lost out to much less capable but cheaper general purpose hardware. There's no point in designing fancy machines unless there's some prospect that they can be mass produced and widely used, because otherwise they will be too expensive to be practical; which presumes not only that they have the potential to be widely used, but also that you (or someone else related to the project) is able to communicate that potential to people with enough money to back the project.
# Hardware for Post Scarcity software
Before going forward with this argument, lets go back. Let's go back to the idea of the Clojure map function. In fact, let's go back to the idea of a function.
If a processor is computing a function, and that function has an argument, then before the function can be computed the value of the argument must be computed; and, as the function cannot be computed until the value of the argument has been computed, there is no point in handing off the processing of the argument to another processor, because the first processor will then necessarily be idle until the value is returned. So it may just as well recurse up the stack itself.
However, if a function has two arguments and values of both must be computed, then if the first processor can hand off processing of one of them to another, similar, processor, potentially the two can be processed in the time in which the original processor could process just one. Provided, that is, that the cost of handing off processing to another processor is substantially less than the cost of evaluating the argument - which is to say, as a general thing, the closer one can get the cost of handing off to another processor to the cost of allocating a stack frame on the current processor, the better. And this is where current-generation hardware is losing out: that cost of handing off is just way too high.
Suppose, then, that our processor is a compute node in a Connection-Machine-like hypercube, able to communicate directly at high speed with twenty close neighbours (I'll come back to this point in detail later). Suppose also that each neighbour-connection has a 'busy' line, which the neighbour raises when it is itself busy. So our processor can see immediately without any need for doing a round-robin which of its neighbours are available to do new work.
Our processor receives a function call with seven arguments, each of which is a further function call. It hands six of these off to idle neighbours, pushes one onto its own local stack, computes it, and recurses back to the original stack frame, waits for the last of the other six to report back a value, and then carries on with its processing.
The fly in the ointment here is memory access. I assume all the processors have significant read-only cache (they don't need read-write cache, we're dealing with immutable data; and they only need a very small amount of scratchpad memory). If all six of the other processors find the data they need (for these purposes the executable function definition is also data) in local cache, all is good, and this will be very fast. But what if all have cache misses, and have to request the data from main memory?
This comes down to topology. I'm not at all clear how you even manage to have twenty separate data channels from a single node. To have a data channel from each node, separately, to main memory simply isn't possible - not if you're dealing with very large numbers of compute nodes. So the data bus has to be literally a bus, available to all nodes simultaneously. Which means, each node that wants some data from main memory must ask for it, and then sit watching the bus, waiting for it to be delivered. Which also means that as data is sent out on the bus, it needs to be tagged with what data it is.
## Looking for the future in the past, part two
In talking about the Connection Machine which lurked in the basement of Logica's central London offices, I mentioned that it lurked in a lab where one of the [Xerox 1108 Dandelions](http://en.wikipedia.org/wiki/Interlisp) I was employed to work on was also located. The Dandelion was an interesting machine in itself. In typical computers - typical modern computers, but also typical computers of thirty years ago - the microcode has virtually the status of hardware. While it may technically be software, it is encoded immutably into the chip when the chip is made, and can never be changed.
The Dandelion and its related machines weren't like that. Physically, the Dandelion was identical to the Star workstations which Xerox then sold for very high end word processing. But it ran different microcode. You could load the microcode; you could even, if you were very daring, write your own microcode. In its Interlisp guise, it had all the core Lisp functions as single opcodes. It had object oriented message passing - with full multiple inheritance and dynamic selector-method resolution - as a single opcode. But it also had another very interesting instruction: [BITBLT](http://en.wikipedia.org/wiki/Bit_blit), or 'Bit Block Transfer'.
This opcode derived from yet another set, that developed for an earlier version of the same processor on which Smalltalk was first implemented. It copied an arbitrary sized block of bits from one location in memory to another location in memory, without having to do any tedious and time consuming messing about with incrementing counters (yes, of course counters were being incremented underneath, but they were in registers only accessible to the the microcode and which ran, I think, significantly faster than the 'main' registers). This highly optimised block transfer routine allowed a rich and responsive WIMP interface on a large bitmapped display on what weren't, underneath it all, actually terribly powerful machines.
## BITBLT for the modern age
Why is BITBLT interesting to us? Well, if we can transfer the contents of only one memory location over the bus in a message, and every message also needs a start-of-message marker and an object reference, then clearly the bus is going to run quite slowly. But if we can say, OK, here's an object which comprises this number of words, coming sequentially after this header, then the amount of overhead to queuing messages on the bus is significantly reduced. But, we need not limit ourselves to outputting as single messages on the bus, data which was contiguous in main memory.
Most of the things which will be requested will be either vectors (yes, Java fans, an object is a vector) or lists. Vectors will normally point to other objects which will be needed at the same time as the vector itself is needed; list structures will almost always do so. Vectors will of course normally be contiguous in memory but the things they point to won't be contiguous with them; lists are from this point of view like structures of linked vectors such that each vector has only two cells.
So we can envisage a bus transfer language which is in itself like a very simple lisp, except decorated with object references. So we might send the list '(1000 (2000) 3000) over the bus as notionally
[ #00001 1000 [ #00002 2000 ] 3000 ]
where '[' represents start-of-object, '#00001' is an object reference, '1000' is a numeric value, and ']' is end-of-object. How exactly is this represented on the bus? I'll come back to that; it isn't the main problem just now.
## Requesting and listening
Each processor can put requests onto the 'address bus'. Because the address bus is available to every processing node, every processing node can listen to it. And consequently every processing node does listen to it, noting every request that passes over the bus in a local request cache, and removing the note when it sees the response come back over the data bus.
When a processing node wants a piece of data, it first checks its local memory to see whether it already has a copy. If it does, fine, it can immediately process it. If not, it checks to see whether the piece of data has already been requested. If it has not, it requests it. Then it waits for it to come up the bus, copies it off into local store and processes it.
That all sounds rather elaborate, doesn't it? An extremely expensive way of accessing shared storage?
Well, actually, no. I think it's not. Let's go back to where we began: to map.
Mapping is a very fundamental computing operation; it's done all the time. Apply this same identical function to these closely related arguments, and return the results.
So, first processor gets the map, and passes a reference to the function and arguments, together with indices indicating which arguments to work on, to each of its unemployed neighbours. One of the neighbours then makes a request for the function and the list of arguments. Each other processor sees the request has been made, so just waits for the results. While waiting, each in this second tier of processors may sub-partition its work block and farm out work to unemployed third tier neighbours, and so on. As the results come back up the bus, each processor takes its local copy and gets on with its partition, finally passing the results back to the neighbour who originally invoked it.
## The memory manager
All this implies that somewhere in the centre of this web, like a fat spider, there must be a single agent which is listening on the address bus for requests for memory objects, and fulfilling those requests by writing the objects out to the data bus. That agent is the memory manager; it could be software running on a dedicated processor, or it could be hardware. It really doesn't matter. It's operating a simple fundamental algorithm, maintaining a garbage collected heap of memory items and responding to requests. It shouldn't be running any 'userspace' code.
Obviously, there has to be some way for processor nodes to signal to the memory manager that they want to store new persistent objects; there needs to be some way of propagating back which objects are still referenced from code which is in play, and which objects are no longer referenced and may be garbage collected. I know I haven't worked out all the details yet. Furthermore, of course, I know that I know virtually nothing about hardware, and have neither the money nor the skills to build this thing, so like my enormous game engine which I really know I'll never finish, it's really more an intellectual exercise than a project.
But... I do think that somewhere in these ideas there are features which would enable us to build higher performance computers which we could actually program, with existing technology. I wouldn't be surprised to see systems fairly like what I'm describing here becoming commonplace within twenty years.
\[Note to self: when I come to rework this essay it would be good to reference [Steele and Sussman, Design of LISP-based Processors](http://repository.readscheme.org/ftp/papers/ai-lab-pubs/AIM-514.pdf).\]

261
docs/Post-scarcity-software.md
View file

@ -1,261 +0,0 @@
# Post Scarcity Software
_This is the text of my essay Post-scarcity Software, originally published in 2006 on my blog [here](https://www.journeyman.cc/blog/posts-output/2006-02-20-postscarcity-software/)._
For years we've said that our computers were Turing equivalent, equivalent to Turing's machine U. That they could compute any function which could be computed. They aren't, of course, and they can't, for one very important reason. U had infinite store, and our machines don't. We have always been store-poor. We've been mill-poor, too: our processors have been slow, running at hundreds, then a few thousands, of cycles per second. We haven't been able to afford the cycles to do any sophisticated munging of our data. What we stored - in the most store intensive format we had - was what we got, and what we delivered to our users. It was a compromise, but a compromise forced on us by the inadequacy of our machines.
The thing is, we've been programming for sixty years now. When I was learning my trade, I worked with a few people who'd worked on Baby - the Manchester Mark One - and even with two people who remembered Turing personally. They were old then, approaching retirement; great software people with great skills to pass on, the last of the first generation programmers. I'm a second generation programmer, and I'm fifty. Most people in software would reckon me too old now to cut code. The people cutting code in the front line now know the name Turing, of course, because they learned about U in their first year classes; but Turing as a person - as someone with a personality, quirks, foibles - is no more real to them than Christopher Columbus or Noah, and, indeed, much less real than Aragorn of the Dunedain.
In the passing generations we've forgotten things. We've forgotten the compromises we've made; we've forgotten the reasons we've made them. We're no longer poor. The machine on which I'm typing this - my personal machine, on my desk, used by no-one but me - has the processor power of slightly over six thousand DEC VAXes; it has the one hundred and sixty two thousand times as much core store as the ICL 1900 mainframe on which I learned Pascal. Yet both the VAX and the 1900 were powerful machines, capable of supporting dozens of users at the same time. Compared to each individual user of the VAX, of the 1900, I am now incalculably rich. Vastly. Incomprehensibly.
And it's not just me. With the exception of those poor souls writing embedded code for micro-controllers, every programmer now working has processor and store available to him which the designers of the languages and operating systems we still use could not even have dreamed of. UNIX was designed when 32 bit machines were new, when 16,384 bytes was a lot of memory and very expensive. VMS - what we now call 'Windows XP' - is only a few years younger.
The compromises of poverty are built into these operating systems, into our programming languages, into our brains as programmers; so deeply ingrained that we've forgotten that they are compromises, we've forgotten why we chose them. Like misers counting grains on the granary floor while outside the new crop is falling from the stalks for want of harvesting, we sit in the middle of great riches and behave as though we were destitute.
One of the things which has made this worse in recent years is the rise of Java, and, following slavishly after it, C#. Java is a language which was designed to write programs for precisely those embedded micro-controllers which are still both store and mill poor. It is a language in which the mind-set of poverty is consciously ingrained. And yet we have adopted it as a general purpose programming language, something for which it is not at all suitable, and in doing so have taught another generation of programmers the mind-set of poverty. Java was at least designed; decisions were made for reasons, and, from the point of view of embedded micro-controllers, those reasons were good. C# is just a fit of pique as software. Not able to 'embrace and extend' Java, Microsoft aped it as closely as was possible without breaching Sun's copyright. Every mistake, every compromise to poverty ingrained in Java is there in C# for all the world to see.
It's time to stop this. Of course we're not as wealthy as Turing. Of course our machines still do not have infinite store. But we now have so much store - and so many processor cycles - that we should stop treating them as finite. We should program as if we were programming for U.
# Store, Name and Value
So let's start with what we store, what we compute on: values. For any given column within a table, for every given instance variable in a class, every record, every object is constrained to have a value with a certain format.
This is, of course, historical. Historically, when storage was expensive we stored textual values in fields of fixed width to economise on storage; we still do so largely because that's what we've always done rather than because there's any longer any rational reason to. Historically, when storage and computation were expensive, we stored numbers in twos-complement binary strings in a fixed number of bytes. That's efficient, both of store and of mill.
But it is no longer necessary, nor is it desirable, and good computer languages such as LISP transparently ignores the difference between the storage format of different numbers. For example:
(defun factorial (n)
(cond
((eq n 1) 1)
(t (* n (factorial (- n 1))))))
;; a quick way to generate very big numbers...
We can add the value of factorial 100 to an integer, say 2, in just the same way that we can add any other two numbers:
(+ (fact 100) 2)
933262154439441526816992388562667004907159682643816214685929638952175999932299156089414639761565182862536979208272237582511852109168 64000000000000000000000002
We can multiply the value of factorial 100 by a real number, say pi, in just the same way as we can add any other two numbers:
(* (factorial 100) pi)
2.931929528260332*10^158
The important point to note here is that there's no explicit call to a bignum library or any other special coding. LISP's arithmetic operators don't care what the underlying storage format of a number is, or rather, are able transparently to handle any of the number storage formats - including bignums - known to the system. There's nothing new about this. LISP has been doing this since the late 1960s. Which is as it should be, and, indeed, as it should be in storage as well as in computation.
A variable or a database field (I'll treat the two as interchangeable, because, as you will see, they are) may reasonably have a validation rule which says that a value which represents the longitude of a point on the Earth in degrees should not contain a value which is greater than 360. That validation rule is domain knowledge, which is a good thing; it allows the system to have some vestige of common sense. The system can then throw an exception when it is asked to store 764 as the longitude of a point, and this is a good thing.
Why then should a database not throw an exception when, for example, a number is too big to fit in the internal representation of a field? To answer, here's a story I heard recently, which seems to be apocryphal, but which neatly illustrates the issue just the same.
_The US Internal Revenue Service have to use a non-Microsoft computer to process Bill Gate's income tax, because Microsoft computers have too small an integer representation to represent his annual income._
Twos complement binary integers stored in 32 bits can represent plus or minus 2,147,483,648, slightly over two US billion. So it's easily possible that Bill Gates' income exceeds this. Until recently, Microsoft operating systems ran only on computers with a register size of 32 bits. Worryingly, the default integer size of my favourite database, Postgres, is also 32 bits.
This is just wrong. Nothing in the domain of income places any fixed upper bound on the income a person may receive. Indeed, with inflation, the upper limit on incomes as quantity is likely to continue to rise. Should we patch the present problem by upping the size of the integer to eight bytes?
In Hungary after the end of World War II inflation ran at 4.19 * 10^16 percent per month - prices doubled every 15 hours. Suppose Gates' income in US dollars currently exceeds the size of a thirty two bit integer, it would take at most 465 hours - less than twenty days - to exceed US$9,223,372,036,854,775,808. What's scary is how quickly you'd follow him. If your present annual salary is just thirty three thousand of your local currency units, then given that rate of inflation, you would overflow a sixty-four bit integer in just 720 hours, or less than a month.
Lots of things in perfectly ordinary domains are essentially unbounded. They aren't shorts. They aren't longs. They aren't doubles. They're numbers. And a system asked to store a number should store a number. Failure to store a number because it's size violates some constraint derived from domain knowledge is desirable behaviour; failure to store a number because it size violates the internal storage representation of the system is just bad, outdated, obsolete system design. Yes, it's efficient of compute power on thirty-two bit processors to store values in thirty-two bit representations. Equally, it's efficient of disk space for a database to know in advance just how mush disk it has to reserve for each record in a table, so that to skip to the Nth record it merely has to skip forward (N * record-size) bytes.
But we're no longer short of either processor cycles or disk space. For a database to reject a value because it cannot be stored in a particular internal representation is industrial archaeology. It is a primitive and antiquated workaround from days of hardware scarcity. In these days of post-scarcity computing, it's something we should long have forgotten, long have cast aside.
This isn't to say that integers should never be stored in thirty-two bit twos complement binary strings. Of course they should, when it's convenient to do so. It's a very efficient storage representation. Of course, when a number overflows a thirty two bit cell, the runtime system has got to throw an exception, has got to deal with it, and consequently the programmer who writes the runtime system has still got to know about and understand the murky aspects of internal storage formats.
Perhaps the language designer, and the programmer who writes the language compiler should, too, but personally I don't think so. I think that at the layer in the system - the level of abstraction - at which the compiler writer works, the operator 'plus' should just be a primitive. It takes two numbers, and returns a number. That's all. The details of whether that's a float, a double, a rational or a bignum should not be in the least relevant at the level of language. There is a difference which is important between a real number and an integer. The old statistical joke about the average family having 2.4 children is funny precisely because it violates our domain knowledge. No family has 2.4 children. Some things, including children, are discrete, however indiscreet you may think them. They come in integral quantities. But they don't come in short quantities or long quantities. Shorts and longs, floats and doubles are artefacts of scarcity of store. They're obsolete.
From the point of view of the runtime designer, the difference between a quantity that can be stored in two bytes, or four, or eight must matter. From the point of view of the application designer, the language designer, even the operating system designer, they should disappear. An integer should be an integer, whether it represents the number of toes on your left foot (about 5), the number of stars in the galaxy (about 1x1011) or the number of atoms in the universe (about 1x1079). Similarly, a real number should be just a real number.
This isn't to say we can't do data validation. It isn't to say we can't throw a soft exception - or even a hard one - when a value stored in a variable or field violates some expectation, which may be an expectation about size. But that should be an expectation based on domain knowledge, and domain knowledge alone; it should not be an expectation based on implementation knowledge.
Having ranted now for some time about numbers, do you think I'm finished? I'm not. We store character values in databases in fields of fixed size. How big a field do we allocate for someone's name? Twenty four characters? Thirty-two? We've all done it. And then we've all found a person who violates our previous expectation of the size of a name, and next time we've made the field a little bigger. But by the time we've made a field big enough to store Charles Philip Arthur George Windsor or Sirimavo Ratwatte Dias Bandaranaike we've negated the point of fixed width fields in the first place, which was economy. There is no natural upper bound to the length of a personal name. There is no natural upper bound to the length of a street address. Almost all character data is a representation at some level of things people say, and the human mind doesn't work like that.
Of course, over the past fifty years, we've tried to make the human mind work like that. We've given addresses standardised 'zip codes' and 'postcodes', we've given people standardised 'social security numbers' and 'identity codes'. We've tried to fit natural things into fixed width fields; we've tried to back-port the inadequacies of our technology onto the world. It's stupid, and it's time we stopped.
So how long is a piece of string? How long is a string of characters? It's unbounded. Most names are short, because short names are convenient and memorable. But that does not mean that for any given number of characters, it's impossible that there should be something with a normal name of that length. And names are not the only things we store in character strings. In character strings we store things people say, and people talk a lot.
At this point the C programmers, the Java programmers are looking smug. Our strings, they say, are unbounded. Sorry lads. A C string is a null terminated sequence of bytes. It can in principle be any length. Except that it lives in a malloced lump of heap (how quaint, manually allocating store) and the maximum size of a lump of heap you can malloc is size_t, which may be 231, 232, 263 or 264 depending on the system. Minus one, of course, for the null byte. In Java, similarly, the size of a String is an int, and an int, in Java, means 231.
Interestingly, Paul Graham, in his essay 'The Hundred YearLanguage', suggests doing away with stings altogether, and representing them as lists of characters. This is powerful because strings become S-expressions and can be handled as S-expressions; but strings are inherently one-dimensional and S-expressions are not. So unless you have some definite collating sequence for a branching 'string' it's meaning may be ambiguous. Nevertheless, in principle and depending on the internal representation of a CONS cell, a list of characters can be of indefinite extent, and, while it isn't efficient of storage, it is efficient of allocation and deallocation; to store a list of N characters does not require us to have a contiguous lump of N bytes available on the heap; nor does it require us to shuffle the heap to make a contiguous lump of that size available.
So; to reprise, briefly.
A value is just a value. The internal representation of a value is uninteresting, except to the designer and author of the runtime system - the virtual machine. For programmers at every other level the internal representation of every value is DKDC: don't know, don't care. This is just as true of things which are fundamentally things people say, things which are lists and things which are pools, as it is of numbers. The representation that the user - including the programmer - deals with is the representation which is convenient and comfortable. It does not necessarily have anything to do with the storage representation; the storage representation is something the runtime system deals with, and that the runtime system effectively hides. Operators exposed by the virtual machine are operators on values. It is a fundamental error, a failure of the runtime designer's most basic skill and craft, for a program ever to fail because a value could not be represented in internal representation - unless the store available to the system is utterly exhausted.
# Excalibur and the Pool
A variable is a handle in a namespace; it gives a name to a value, so that we can recall it. Storing a value in a variable never causes an exception to be thrown because the value cannot be stored. But it may, reasonably, justifiably, throw an exception because the value violates domain expectations. Furthermore, this exception can be either soft or hard. We might throw a soft exception if someone stored, in a variable representing the age of a person in years, the value 122. We don't expect people to reach one hundred and twenty two years of age. It's reasonable to flag back to whatever tried to set this value that it is out of the expected range. But we should store it, because it's not impossible. If, however, someone tries to store 372 in a variable representing longitude in degrees, we should throw a hard exception and not store it, because that violates not merely a domain expectation but a domain rule.
So a variable is more than just a name. It is a slot: a name with some optional knowledge about what may reasonably be associated with itself. It has some sort of setter method, and possibly a getter method as well.
I've talked about variables, about names and values. Now I'll talk about the most powerful abstraction I use - possibly the most powerful abstraction in software - the namespace. A namespace is a sort of pool into which we can throw arbitrary things, tagging each with a distinct name. When we return to the pool and invoke a name, the thing in the pool to which we gave that name appears.
## Regularities: tables, classes, patterns
Database tables, considered as sets of namespaces, have a special property: they are regular. Every namespace which is a record in the same table has the same names. A class in a conventional object oriented language is similar: each object in the class has the same set of named instance variables. They match a pattern: they are in fact constrained to match it, simply by being created in that table or class.
Records in a table, and instance variables in a class, also have another property in common. For any given name of a field or instance variable, the value which each record or object will store under that name is of the same type. If 'Age' is an integer in the definition of the table or class, the Age of every member will be an integer. This property is different from regularity, and, lacking a better word for it, I'll call it homogeneity. A set of spaces which are regular (i.e. share the same names) need not be homogeneous (i.e. share the same value types for those names), but a set which is homogeneous must be regular.
But records in a table, in a view, in a result set are normally in themselves values whose names are the values of the key field. And the tables and views, too, are values in a namespace whose names are the table names, and so on up. Namespaces, like Russian dolls, can be nested indefinitely. By applying names to the nested spaces at each level, we can form a path of names to every space in the meta-space and to each value in each space, provided that the meta-space forms an acyclic directed graph (this is, after all, the basis of the XPath language. Indeed, we can form paths even if the graph has cycles, provided every cycle in the graph has some link back to the root.
## Social mobility
It's pretty useful to gather together all objects in the data space which match the same pattern; it's pretty useful for them all to have distinct names. So the general concept of a regularity which is itself a namespace is a useful one, even if the names have to be gensymed.
To be in a class (or table), must a space be created in that class (or table)? I don't see why. One of my earlier projects was an inference engine called Wildwood, in which objects inferred their own class by exploring the taxonomy of classes until they found the one in which they felt most comfortable. I think this is a good model. You ought to be able to give your dataspace a good shake and then pull out of it as a collection all the objects which match any given pattern, and this collection ought to be a namespace. It ought to be so even if the pattern did not previously exist in the data space as the definition of a table or class or regularity or whatever you care to call it.
A consequence of this concept is that objects which acquire new name-value pairs may move out of the regularity in which they were created either to exist as stateless persons in the no-man's land of the dataspace, or into a new regularity; or may form the seed around which a new regularity can grow. An object which acquires a value for one of its names which violates the validation constraints of one homogeneity may similarly move out into no-man's land or into another. In some domains, in some regularities, it may be a hard error to do this (i.e. the system will prevent it). In some domains, in some regularities, it may be a soft error (i.e. the system allows it under protest). In some domains, in some regularities, it may be normal; social mobility of objects will be allowed.
## Permeability
There's another feature of namespaces which gets hard wired into lots of software structures without very often being generalised, and that is permeability, semi-translucency. In my toolkit Jacquard, for example, values are first searched for in the namespace of http parameters; if not found there, in the namespace of cookies; next, in the namespace of session variables, then in local configuration parameters, finally in global configuration parameters. There is in effect a layering of semi-translucent namespaces like the veils of a dancer.
It's not a pattern that's novel or unique to Jacquard, of course. But in Jacquard it's hard wired and in all the other contexts in which I've seen this pattern it's hardwired. I'd like to be able to manipulate the veils; to add, or remove, of alter the layering. I'd like this to be a normal thing to be able to do.
The Name of the Rose: normativeness and hegemony
I have a friend called Big Nasty. Not everyone, of course, calls him Big Nasty. His sons call him 'Dad'. His wife calls him 'Norman'. People who don't know him very well call him 'Mr Maxwell'. He does not have one true name.
The concept of a true name is a seductive one. In many of the traditions of magic - and I have always seen software as a technological descendant or even a technological implementation of magic - a being invoked by its true name must obey. In most modern programming languages, things tend to have true names. There is a protocol for naming Java packages which is intended to guarantee that every package written anywhere in the world has a globally unique true name. Globally unique true names do then have utility. It's often important when invoking something to be certain you know exactly what it is you're invoking.
But it does not seem to me that this hegemonistic view of the dataspace is required by my messy conception. Certainly it cannot be true that an object has only one true name, since it may be the value of several names within several spaces (and of course this is true of Java; a class well may have One True Name, but I can still create an instance variable within an object whose name is anythingILike, and have its value is that class).
The dataspace I conceive is a soup. The relationships between regularities are not fixed, and so paths will inevitably shift. And in the dataspace, one sword can be in many pools - or even many times in the same pool, under different names - at the same time. We can shake the dataspace in different ways to see different views on the data. There should be no One True hegemonistic view.
This does raise the question, 'what is a name'. In many modern relational databases, all primary keys are abstract and are numbers, even if natural primary keys exist in the data - simply because it is so easy to create a table with an auto-incrementer on the key field. Easy, quick, convenient, lazy, not always a good thing. In terms of implementation details, namespaces are implemented on top of hash tables, and any data object can be hashed. So can anything be a name?
In principle yes. However, my preference would be to purely arbitrarily say no. My preference would be to say that a name must be a 'thing people say', a pronounceable sequence of characters; and also, with no specific upper bound, reasonably short.
## The Problem with Syntax
Let me start by saying that I really don't understand the problem with syntax. Programming language designers spend a lot of time worrying about it, but I believe they're simply missing the point. People say 'I can't learn LISP because I couldn't cope with all the brackets'. People - the Dylan team, for one - have developed systems which put a skin of 'normal' (i.e., ALGOL-like) syntax on top of LISP. I personally won't learn Python because I don't trust a language where white space is significant. But in admitting that prejudice I'm admitting to a mistake which most software people make.
We treat code as if it wasn't data. We treat code as if it were different, special. This is the mistake made by the LISP2 brigade, when they gave their LISPs (ultimately including Common LISP) separate namespaces, one for 'code' and one for 'data'. It's a fundamental mistake, a mistake which fundamentally limits our ability to even think about software.
What do I mean by this?
Suppose I ask my computer to store pi, 3.14159265358979. Do I imagine that somewhere deep within the machine there is a bitmap representation of the characters? No, of course I don't. Do I imagine there's a vector starting with the bytes 50 46 49 51 49 53 57 ...? Well, of course, there might be, but I hope there isn't because it would be horribly inefficient. No, I hope and expect there's an IEEE 754 binary encoding of the form 01100100100001111...10. But actually, frankly, I don't know, and I don't care, provided that it is stored and that it can be computed with.
However, as to what happens if I then ask my computer to show me the value it has stored, I do know and I do care. I expect it to show me the character string '3.14159265358979' (although I will accept a small amount of rounding error, and I might want it to be truncated to a certain number of significant figures). The point is, I expect the computer to reflect the value I have stored back to me in a form which it is convenient for me to read, and, of course, it can.
We don't, however, expect the computer to be able to reflect back an executable for us in a convenient form, and that is in itself a curious thing. If we load, for example, the UNIX command 'ls' into a text editor, we don't see the source code. We see instead, the raw internal format. And the amazing thing is that we tolerate this.
It isn't even that hard to write a 'decompiler' which can take a binary and reflect back source code in a usable form. Here, for example, is a method I wrote:
/**
* Return my action: a method, to allow for specialisation. Note: this
* method was formerly 'getAction()'; it has been renamed to disambiguate
* it from 'action' in the sense of ActionWidgets, etc.
*/
public String getNextActionURL( Context context ) throws Exception
{
String nextaction = null;
HttpServletRequest request =
(HttpServletRequest) context.get( REQUESTMAGICTOKEN );
if ( request != null )
{
StringBuffer myURL = request.getRequestURL( );
if ( action == null )
{
nextaction = myURL.toString( );
// If I have no action, default my action
// to recall myself
}
else
{
nextaction =
new URL( new URL( myURL.toString( ) ), action ).toString( );
// convert my action into a fully
// qualified URL in the context of my
// own
}
}
else
{ // should not happen!
throw new ServletException( "No request?" );
}
return nextaction;
}
and here is the result of 'decompiling' that method with an open-source Java decompiler, jreversepro:
public String getNextActionURL(Context context)
throws Exception
{
Object object = null;
HttpServletRequest httpservletrequest =
(HttpServletRequest)context.get( "servlet_request");
String string;
if (httpservletrequest != null) {
StringBuffer stringbuffer = httpservletrequest.getRequestURL();
if (action == null)
string = stringbuffer.toString();
else
string = new URL(new URL(stringbuffer.toString()) ,
action).toString();
}
else
throw new ServletException("No request?");
return (string);
}
As you can see, the comments have been lost and some variable names have changed, but the code is essentially the same and is perfectly readable. And this is with an internal form which has not been designed with decompilation in mind. If decompilation had been designed for in the first place, the binary could have contained pointers to the variable names and comments. Historically we haven't done this, both for 'intellectual property' reasons and because of store poverty. In future, we can and will.
Again, like so much in software, this isn't actually new. The microcomputer BASICs of the seventies and eighties 'tokenised' the source input by the user. This tokenisation was not of course compilation, but it was analogous to it. The internal form of the program that was stored was much terser then the representation the user typed. But when the user asked to list the program, it was expanded into its original form.
Compilation - even compilation into the language of a virtual machine - is much more sophisticated than tokenising, of course. Optimisation means that many source constructs may map onto one object construct, and even that one source construct may in different circumstances map onto many object constructs. Nevertheless it is not impossible - nor even hugely difficult - to decompile object code back into readable, understandable and editable source.
But Java syntax is merely a format. When I type a date into a computer, say '05-02-2005', and ask it to reflect that date back to me, I expect it to be able to reflect back to me '05-02-2006'. But I expect it to be able to reflect back to an American '02-05-2006', and to either of us 'Sunday 5th February 2006' as well. I don't expect the input format to dictate the output format. I expect the output format to reflect the needs and expectations of the person to whom it is displayed.
To summarise, again.
Code is data. The internal representation of data is Don't Know, Don't Care. The output format of data is not constrained by the input format; it should suit the use to which it is to be put, the person to whom it is to be displayed.
Thus if the person to whom my Java code is reflected back is a LISP programmer, it should be reflected back in idiomatic LISP syntax; if a Python programmer, in idiomatic Python syntax. Let us not, for goodness sake, get hung up about syntax; syntax is frosting on the top. What's important is that the programmer editing the code should edit something which is clearly understandable to him or her.
This has, of course, a corollary. In InterLISP, one didn't edit files 'out of core' with a text editor. One edited the source code of functions as S-expressions, in core, with a structure editor. The canonical form of the function was therefore the S-expression structure, and not the printed representation of it. If a piece of code - a piece of executable binary, or rather, of executable DKDC - can be reflected back to users with a variety of different syntactic frostings, none of these can be canonical. The canonical form of the code, which must be stored in version control systems or their equivalent, is the DKDC itself; and to that extent we do care and do need to know, at least to the extent that we need to know that the surface frosting can again be applied systematically to the recovered content of the archive.
# If God does not write LISP
I started my professional life writing LISP on Xerox 1108s and, later, 1186s - Dandelions and Daybreaks, if you prefer names to numbers. When I wanted to multiply two numbers, I multiplied two numbers. I didn't make sure that the result wouldn't overflow some arbitrary store size first. When a function I wrote broke, I edited in its structure in its position on the stack, and continued the computation. I didn't abort the computation, find a source file (source file? How crude and primitive), load it into a text editor, edit the text, save it, check for syntax errors, compile it, load the new binary, and restart the computation. That was more than twenty years ago. It is truly remarkable how software development environments have failed to advance - have actually gone backwards - in that time.
LISP's problem is that it dared to try to behave as though it were a post-scarcity language too soon. The big LISP machines - not just the Xerox machines, the LMI, Symbolics, Ti Explorer machines - were vastly too expensive. My Daybreak had 8Mb of core and 80Mb of disk when PCs usually didn't even have the full 640Kb. They were out-competed by UNIX boxes from Sun and Apollo, which delivered less good software development environments but at a much lower cost. They paid the price for coming too early: they died. And programmers have been paying the price for their failure ever since.
But you only have to look at a fern moss, a frond of bracken, an elm sapling, the water curling over the lip of a waterfall, to know that if God does not write LISP She writes some language so similar to LISP as to make no difference. DNA encodes recursive functions; turbulent fluids move in patterns formed by recursion, whorls within whorls within whorls.
The internal structure, then, of the post scarcity language is rather lisp-like. Don't get hung up on that! Remember that syntax isn't language, that the syntax you see need not be the syntax I see. What I mean by saying the language is lisp-like is that its fundamental operation is recursion, that things can easily be arranged into arbitrary structures, that new types of structure can be created on the fly, that new code (code is just data, after all) can be created and executed on the fly, that there is no primacy of the structures and the code created by the programmer over the structures and code created by the running system; that new code can be loaded and linked seamlessly into a running system at any time. That instead of little discrete programs doing little discrete specialised things in separate data spaces each with its own special internal format and internal structures, the whole data space of all the data available to the machine (including, of course, all the code owned by the machine) exists in a single, complex, messy, powerful pool. That a process doesn't have to make a special arrangement, use a special protocol, to talk to another process or to exchange data with it.
In that pool, the internal storage representation of data objects is DKDC. We neither have nor need to have access to it. It may well change over time without application layer programs even being aware or needing to be aware of the change, certainly without them being recompiled.
The things we can store in the dataspace include:
1. **integers** of any size
1. **reals** to any appropriate degree of precision
1. **rationals, complex numbers**, and other things we might want to compute with
1. **dates, times**, and other such useful things
1. **things people say** of any extent, from names to novels
1. **lists of any extent**, branching or not
1. **slots** associations of names with some setter and, perhaps, getter knowledge which determine what values can be stored under that name
1. **namespaces** collections, extensible or not, of slots
1. **regularities** collections of namespaces each of which share identical names
1. **homogeneities** collections of namespaces each of which share identical slots
1. **functions** all executable things are 'functions' in a lispy sense. They are applied to arguments and return values. They may or may not have internal expectations as to the value type of those arguments.
1. **processes** I don't yet have a good feeling for what a post-scarcity process looks like, at top level. It may simply be a thread executing a function; I don't know. I don't know whether there needs to be one specially privileged executive process.
Things which we no longer store - which we no longer store because they no longer have any utility - include
1. **shorts, longs, doubles**, etc specific internal representation types. You saw that coming.
1. **tables**, and with them, **relational databases** and **relational database management systems** no longer needed because the pool is itself persistent (although achieving the efficiency of data access that mature RDBMS give us may be a challenge).
1. **files** You didn't see that coming?
Files are the most stupid, arbitrary way to store data. Again, with a persistent data pool, they cease to have any purpose. Post scarcity, there are no files and there is no filesystem. There's no distinction between in core and out of core. Or rather, if there are files and a filesystem, if there is a distinction between in core and out of core, that distinction falls under the doctrine of DKDC: we don't know about it, and we don't care about it. When something in the pool wants to use or refer to another something, then that other something is available in the pool. Whether it was there all along, or whether it was suddenly brought in from somewhere outside by the runtime system, we neither know nor care. If things in the pool which haven't been looked at for a long time are sent to sulk elsewhere by the runtime system that is equally uninteresting. Things which are not referenced at all, of course, may be quietly dropped by the runtime system in the course of normal garbage collection.
One of the things we've overloaded onto the filesystem is security. In core, in modern systems, each process guards its own pool of store jealously, allowing other processes to share data with it only through special channels and protocols, even if the two processes are run by the same user identity with the same privilege. That's ridiculous. Out of core, data is stored in files often with inscrutable internal format, each with its own permissions and access control list.
It doesn't need to be that way. Each primitive data item in core - each integer, each list node, each slot, each namespace - can have its own access control mechanism. Processes, as such, will never 'own' data items, and will certainly never 'own' chunks of store - at the application layer, even the concept of a chunk of store will be invisible. A process can share a data item it has just created simply by setting an appropriate access policy on it, and programmers will be encouraged normally to be as liberal in this sharing as security allows. So the slot Salary of the namespace Simon might be visible only to the user Simon and the role Payroll, but that wouldn't stop anyone else looking at the slot Phone number of the same namespace.
Welcome, then, to post scarcity computing. It may not look much like what you're used to, but if it doesn't it's because you've grown up with scarcity, and even since we left scarcity behind you've been living with software designed by people who grew up with scarcity, who still hoard when there's no need, who don't understand how to use wealth. It's a richer world, a world without arbitrary restrictions. If it looks a lot like Alan Kay (and friends)'s Croquet, that's because Alan Kay has been going down the right path for a long time.

20
docs/Regularity.md
View file

@ -1,20 +0,0 @@
# Regularity
A regularity is a map whose values are maps, all of whose members share the same keys. A map may be added to a regularity only if it has all the keys the regularity expects, although it may optionally have more. It is legitimate for the same map to be a member of two different regularities, if it has a union of their keys. Keys in a regularity must be keywords. Regularities are roughly the same sort of thing as classes in object oriented programming or tables in databases, but the values of the keys are not policed (see homogeneity).
A regularity may also have an association of methods, that is, functions which accept a member of the regularity as their first argument; this set of methods forms an API to the regularity. Of course a full hierarchical object oriented model can be layered on top of this, but a regularity does not in itself have any concept of class inheritance.
But, for example, if we have a regularity whose members represent companies, and those companies each have employees, then there might be a method :payroll of companies which might internally look like:
(lambda (company)
(reduce + (map do-something-to-get-salary (:employees company))))
which would be accessed
(with ((companies . ::shared:pool:companies)
(acme . companies:acme-widgets))
(companies:methods:payroll acme))
But salary is not a property of a company, it's a property of an employee; so what is this thing called do-something-to-get-salary? It's a method on the regularity of employees, so in this example, it is ::shared:pool:employees:methods:salary.
There are issues that I haven't resolved yet about the mutability of regularities and homogeneities; obviously, in order to provide multi-user visibility of current values of shared data, some regularities must be mutable. But mutability has potentially very serious perfomance issues for the hypercube architecture, so I think that in general they should not be.

40
docs/Stack.md
View file

@ -1,40 +0,0 @@
# Stack
The C (and I assume but don't know) Rust stack are contiguous blocks of memory which grow down from the top of the virtual memory map allocated by the operating system to the process. The Lisp stack doesn't have to be the same as the C stack and in fact probably cannot be if I want to have multiple Lisp threads running concurrently in the same process.
If the Lisp stack and the implementation language stack are different, then it's more awkward for Lisp to call functions written in the implementation language and vice versa, but not impossible.
Past Lisps have implemented stack as lists and as vectors. Both work. My own guess is that it possibly best to have a standard sized stack frame allocated in vector space, so that each frame is a contiguous block of memory. A stack frame needs to contain parameters, a return pointer, and somewhere the caller will pick up the return value from. I think a stack frame should have the following:
+-----------------+-----------------+---------------------------------------------------+
| tag | 0...31 | 'STCK' |
+-----------------+-----------------+---------------------------------------------------+
| vecp-pointer | 32...95 | cons-pointer to my VECP (or NIL?) |
+-----------------+-----------------+---------------------------------------------------+
| size | 96...159 | 77 |
+-----------------+-----------------+---------------------------------------------------+
| tag | 160...167 | 0 |
+-----------------+-----------------+---------------------------------------------------+
| parameter 1 | 168...231 | cons-pointer to first param |
+-----------------+-----------------+---------------------------------------------------+
| parameter 2 | 232...295 | cons-pointer to second param |
+-----------------+-----------------+---------------------------------------------------+
| parameter 3 | 296...359 | cons-pointer to third param |
+-----------------+-----------------+---------------------------------------------------+
| more params | 360...423 | cons-pointer to list of further params |
+-----------------+-----------------+---------------------------------------------------+
| return pointer | 424...487 | memory address of the instruction to return to |
+-----------------+-----------------+---------------------------------------------------+
| return value | 488...551 | cons pointer to return value |
+-----------------+-----------------+---------------------------------------------------+
| prior frame ptr | 552...615 | cons-pointer to preceding stack frame VECP |
+-----------------+-----------------+---------------------------------------------------+
Note that every argument to a Lisp function must be a [cons space object](Cons-space.html) passed by reference (i.e., a cons pointer). If the actual argument is actually a [vector space](Vector-space.html) object, then what we pass is a reference to the VECP object which references that vector.
I'm not certain we need a prior frame pointer; if we don't, we may not need a VECP pointing to a stack frame, since nothing can point to a stack frame other than the next stack frame(s) up the stack (if we parallelise *map*, *and* and so on) which to implement a multi-thread system we essentially must have, there may be two or more successor frames to any frame. In fact to use a massively multiprocessor machine efficiently we must normally evaluate each parameter in a separate thread, with only special forms such as *cond* which impose explicit control flow evaluating their clauses serially in a single thread.
*Uhhhmmm... to be able to inspect a stack frame, we will need a pointer to the stack frame. Whether that pointer should be constructed when the stack frame is constructed I don't know. It would be overhead for something which would infrequently be used.*
However, modern systems with small numbers of processors and expensive thread construction and tear-down would perform **terribly** if all parameter evaluation was parallelised, so for now we can't do that, even though the semantics must be such that later we can.

19
docs/Sysout-and-sysin.md
View file

@ -1,19 +0,0 @@
# Sysout and sysin
We need a mechanism to persist a running system to backing store, and restore it from backing store.
This might, actually, turn out not to be terribly hard, but is potentially horrendous, particularly if we're talking about very large (multi-terabyte) memory images.
If we use paged memory, as many UNIX systems do, then memory pages periodically get written to disk and the sum total of the memory pages on disk represent an image of the state of system memory. The problem with this is that the state of system memory is changing all the time, and if some pages are out of date with respect to others you don't have a consistent image.
However, the most volatile area of memory is at the outer end of [cons space](Cons-space.html), since that is where cons cells are most likely to die and consequently where new cons cells are most likely to be allocated. We could conceivably take advantage of this by maintaining a per-page [free list](Free-list.html), and preferentially allocating from the currently busiest page. Volatility in [vector space](Vector-space.html) is likely to be significantly lower, but significantly more distributed. However, if we stick to the general rule that objects aren't mutable, volatility happens only by allocating new objects or deallocating old ones. So it may be the case that if we make a practice of flushing vector space pages when that page is written to, and flushing the active cons space pages regularly, we may at any time achieve a consistent memory image on disk even if it misses the last few seconds worth of changes in cons space.
Otherwise it's worth looking at whether we could journal changes between page flushes. This may be reasonably inexpensive.
If none of this works then persisting the system to backing media may mean halting the system, compacting vector space, writing the whole of active memory to a stream, and restarting the system. This is extremely undesirable because it means putting the system offline for a potentially extended period.
-----
Actually, I'm not sure the above works at all. To sysout a running system, you'd have to visit each node in turn and serialise its cons and vector pages. But if the system is still running when you do this, then you would probably end up with an inconsistent sysout. So you'd have to signal all nodes to halt before performing sysout. Further, you could not restore a sysout to a system with a smaller node count, or smaller node memory, to the system dumped.
This is tricky!

14
docs/System-private-functions.md
View file

@ -1,14 +0,0 @@
# System private functions
**actually, I think this is a bad idea &mdash; or at least needs significantly more thought!**
System-private functions are functions private to the system, which no normal user is entitled to access; these functions normally have an [access control](Access-control.html) value of NIL.
# (sys-access-control arg)
System private. Takes one argument. Returns the access control list of its argument.
# (sys-readable arg user)
System private. Takes two arguments. Returns `TRUE` if the first argument is readable by the reader represented by the second argument; else `NIL`.

35
docs/Topology-of-the-hardware-of-the-deep-future.md
View file

@ -1,35 +0,0 @@
![HAL 9000 - a vision of the hardware of the deep future](https://vignette4.wikia.nocookie.net/2001/images/5/59/Hal_console.jpg/revision/latest?cb=20090823025755)In thinking about how to write a software architecture that won't quickly become obsolescent, I find that I'm thinking increasingly about the hardware on which it will run.
In [Post Scarcity Hardware](Post-scarcity-hardware.html) I envisaged a single privileged node which managed main memory. Since then I've come to thing that this is a brittle design which will lead to bottle necks, and that each cons page will be managed by a separate node. So there needs to be a hardware architecture which provides the shortest possible paths between nodes.
Well, actually... from a software point of view it doesn't matter. From a software point of view, provided it's possible for any node to request a memory item from any other node, that's enough, and, for the software to run (slowly), a linear serial bus would do. But part of the point of this thinking is to design hardware which is orders of magnitude faster than the [von Neumann architecture](https://en.wikipedia.org/wiki/Von_Neumann_architecture) allows. So for performance, cutting the number of hops to a minimum is important.
I've been reading Danny Hillis' [thesis](https://dspace.mit.edu/bitstream/handle/1721.1/14719/18524280-MIT.pdf?sequence=2) and his book [The Connection Machine](https://books.google.co.uk/books/about/The_Connection_Machine.html?id=xg_yaoC6CNEC&redir_esc=y&hl=en) which, it transpires, is closely based on it. Danny Hillis was essentially trying to do what I am trying to do, but forty years ago, with the hardware limitations of forty years ago (but he was trying to do it in the right place, and with a useful amount of money that actually allowed him to build something physical, which I'm never likely to have).
Hillis' solution to the topology problem, as I understand it (and note - I may not understand it very well) is as follows:
![Second generation connection machine in use](http://www.digibarn.com/collections/parts/connection-machine-chip/cm2-500.jpg)
If you take a square grid and place a processor at every intersection, it has at most four proximal neighbours, and, for a grid which is `x` cells in each direction, the longest path between two cells is `2x`. If you join the nodes on the left hand edge of the grid to the corresponding nodes on the right hand edge, you have a cylinder, and the longest path between two nodes is 1.5x. If you then join the nodes on the top of the grid to the nodes on the bottom, you have a torus - a figure like a doughnut or a bagel. Every single node has four proximal neighbours, and the longest path between any two nodes is `x`.
So far so good. Now, let's take square grids and stack them. This gives each node at most six proximal neighbours. We form a cube, and the longest distance between two nodes is `3x`. We can link the nodes on the left of the cube to the corresponding nodes on the right and form a (thick walled) cylinder, and the longest distance between two nodes is `2.5x`. Now join the nodes at the top of the cube to the corresponding nodes at the bottom, and we have a thick walled torus. The maximum distance between is now `2x`.
Let's stop for a moment and think about the difference between logical and physical topology. Suppose we have a printed circuit board with 100 processors on it in a regular grid. We probably could physically bend the circuit board to form a cylinder, but there's no need to do so. We achieve exactly the same connection architecture simply by using wires to connect the left side to the right. And if we use wires to connect those at the top with those at the bottom, we've formed a logical torus even though the board is still flat.
It doesn't even need to be a square board. We could have each processor on a separate board in a rack, with each board having four connectors probably all along the same edge, and use patch wires to connect the boards together into a logical torus.
So when we're converting our cube into a torus, the 'cube' *could* consist of a vertical stack of square boards each of which has a grid of processors on it. But it could also consist of a stack of boards in a rack, each of which has six connections, patched together to form the logical thick-walled torus. So now lets take additional patch leads and join the nodes that had been on the front of the logical cube to the corresponding nodes on the back of the logical cube, and we have a topology which has some of the properties of a torus and some of the properties of a sphere, and is just mind-bending if you try to visualise it.
This shape is what I believe Hillis means by a [hypercube](https://en.wikipedia.org/wiki/Hypercube), although I have to say I've never found any of the visualisations of a hypercube in books or on the net at all helpful, and they certainly don't resemble the torusy-spherey thing I which visualise.
It has the very useful property, however, that the longest distance between any two nodes is `1.5x`.
Why is `1.5x` on the hypercube better than `1x` on the torus? Suppose you want to build a machine with about 1000 nodes. The square root of a thousand is just less than 32, so let's throw in an extra 24 nodes to make it a round 32. We can lay out 1024 nodes on a 32 x 32 square, join left to right, top to bottom, and we have a maximum path between two of 1024 nodes of 32 hops. Suppose instead we arrange our processors on ten boards each ten by ten, with vertical wires connecting each processor with the one above it and the one below it, as well tracks on the board linking each with those east, west, north and south. Connect the left hand side to the right, the front to the back and the top to the bottom, and we have a maximum path between any two of 1000 nodes of fifteen hops. That's twice as good.
Obviously, if you increase the number of interconnectors to each processor above six, the paths shorten further but the logical topology becomes even harder to visualise. This doesn't matter - it doesn't actually have to be visualised - but wiring would become a nightmare.
I've been thinking today about topologies which would allow higher numbers of connections and thus shorter paths, and I've come to this tentative conclusion.
I can imagine topologies which tesselate triangle-tetrahedron-hypertetrahedron and pentagon-dodecahedron-hyperdodecahedron. There are possibly others. But the square-cube-hypercube model has one important property that those others don't (or, at least, it isn't obvious to me that they do). In the square-cube-hypercube model, every node can be addressed by a fixed number of coordinates, and the shortest path from any node to any other is absolutely trivial to compute.
From this I conclude that the engineers who went before me - and who were a lot more thoughtful and expert than I am - were probably right: the square-cube-hypercube model, specifically toruses and hypercubes, is the right way to go.

9
docs/Users.md
View file

@ -1,9 +0,0 @@
# Users
I'm not yet sure what sort of objects users are. They may just be lists, interned in a special namespace such as *system.users*. They may be special purpose [vector space](Vector-space.html) objects (although I don't see why, apart from to get a special tag, which might be useful).
Every user object must contain credentials, and the credentials must be readable by system only; the credentials are either a hashed password or a cryptographic public key. The user object must also have an identifying name, and probably other identifying information. But it's not necessarily the case that every user on the system needs to be able to see the names of every other user on the system, so the identifying information (or the user object itself) may have [access control](Access-control.html) lists.
There is a problem here with the principle of [immutability](Immutability.html); if an access control list on an object _foo_ contains a pointer to my user object so that I can read _foo_, and I change my password, then the immutability rule says that a new copy of the *system.users* namespace is created with a new copy of my user object. This new user object isn't on any access control list so by changing my password I actually can't read anything.
This means that what we put on access control lists is not user objects, but symbols (usernames) which are bound in *system.users* to user objects; the user object then needs a back-pointer to that username. A user group then becomes a list not of user objects but of interned user names.

80
docs/Vector-space.md
View file

@ -1,80 +0,0 @@
# Vector Space
Vector space is what in conventional computer languages is known as 'the heap'. Because objects allocated in vector space are of variable size, vector space will fragment over time. Objects in vector space will become unreferenced, making them available for garbage collection and reallocation; but ultimately you will arrive at the situation where there are a number of small free spaces in vector space but you need a large one. Therefore there must ultimately be a mark-and-sweep garbage collector for vector space.
To facilitate this, reference to every vector space object will be indirected through exactly one VECP object in [cons space](Cons-space.html). If a live vector space object has to be moved in memory in order to compact the heap and to allocate a new object, only one pointer need be updated. This saves enormously on mark-and-sweep time, at the expense of a small overhead on access to vector space objects.
Every vector space object must have a header, indicating that it is a vector space object and what sort of a vector space object it is. Each vector space object must have a fixed size, which is declared in its header. Beyond the header, the payload of a vector space object is undetermined.
Note that, if cons-pointers are implemented simply as memory addresses, the cost of moving a cons page becomes huge, so a rational garbage collector would know about cons pages and do everything possible to avoid moving them.
## The header
As each vector space object has an associated VECP object in cons space, a vector space object does not need to contain either a reference count or an access control list. It does need a cons-pointer to its associated VECP object; it does need a tag (actually it doesn't, since we could put all the tags in cons space, but it is convenient for memory allocation debugging that each should have a tag). It's probably convenient for it to have a mark bit, since if garbage collection of vector space is implemented at all it needs to be mark-and-sweep.
So the header looks like this
+-----+--------------+------+------+--------------+
| tag | vecp-pointer | size | mark | payload... /
+-----+--------------+------+------+------------+
**TODO:** I'm not satisfied with this header design. I think it should be a multiple of 64 bits, so that it is word aligned, for efficiency of fetch. Possibly it would be better to make the *size* field 31 bits with *mark* size one bit, and instead of having the value of *size* being the size of the object in bytes, it should be the size in 64 bit words, even though that makes the maximum allocatable object only 17 gigabytes. It should also be ordered *tag, size, mark, vecp-pointer*, in order to word align the *vecp-pointer* field.
### Tag
The tag will be a 32 bit unsigned integer in the same way and for the same reasons that it is in [cons space](Cons-space.html): i.e., because it will be alternately readable as a four character ASCII string, which will aid memory debugging.
### Vecp-pointer
The vecp pointer is a back pointer to the VECP object in cons space which points to this vector space object. It is, therefore, obviously, the size of a [cons pointer](consspaceobject_8h.html#structcons__pointer), which is to say 64 bits.
### Size
Obviously a single vector space object cannot occupy the whole of memory, since there are other housekeeping things we need to get the system up and running. But there really should not be a reason why a program should not allocate all the remaining available memory as a single object if that's what it wants to do. So the size field should be the width of the address bus of the underlying machine; for the present, 64 bits. The value of the size field will be the whole size, in bytes, of the object including the header.
### Mark
It's probable that in version zero we won't implement garbage collection of vector space. C programs do not normally have any mechanism for compacting their heap; and vector space objects are much less likely than cons space objects to be transient. However, respecting the fact that in the long term we are going to want to be able to compact our vector space, I'll provide a mark field. This really only needs to be one bit, but, again for word alignment, we'll give it a byte.
So the header now looks like this:
+-----+--------------+------+------+------------------------+
| 0 | 32 | 96 | 160 | 168 ...(167 + size) /
| tag | vecp-pointer | size | mark | payload... /
+-----+--------------+------+------+--------------------+
#### Alternative mark-bit strategy
A thought which has recently occurred to me is that the mark bit could be the case bit of the least significant byte of the tag. So that if the tag read 'IMAG' (a raster image), for example, when marked it would read 'IMAg'. This saves having a separately allocated mark in the header, but retains debugging clarity.
## Tags
I really don't at this point have any idea what sorts of things we'll want to store in vector space. This is a non-exhaustive list of things I can think of just now.
### BMAP
A bitmap; a monochrome raster; a two dimensional array of bits.
### EXEC
We definitely need chunks of executable code - compiled functions.
### HASH
We definitely need hashtables. A hashtable is implemented as a pointer to a hashing function, and an array of N cons-pointers each of which points to an [assoc list](Hybrid-assoc-lists.html) acting as a hash bucket. A hashtable is immutable. Any function which 'adds a new key/value pair to' a hashtable in fact returns a new hashtable containing all the key value bindings from the old one, with the new one added. Any function which 'changes a key/value pair' in a hashtable in fact returns a new value with the same bindings of all the keys except the one which has changed as the old one.
In either case, anything which held a pointer to the old version still sees the old version, which continues to exist until everything which pointed to it has been deallocated. Only things which access the hashtable via a binding in a current namespace will see the new version.
### NMSP
A namespace. A namespace is a hashtable with some extra features. It has a parent pointer: NIL in the case of a namespace which was not created by 'adding to' or 'modifying' a pre-existing one, but where a pre-existing one was acted on, then that pre-existing one. It also must have an additional access control list, for users entitled to create new canonical versions of this namespace.
A lot of thinking needs to be done here. It's tricky. If I get it wrong, the cost to either performance or security or both will be horrible.
### RSTR
A raster; a two dimensional array of 32 bit integers, typically interpreted as RGBA colour values.
### VECT
An actual vector; an array with cells of a fixed type (where, obviously, a cons pointer is one type). Has a finite number of dimensions, but probably not more than 4,294,967,296 will be supported (i.e. 32 bits for `dimensions`).

2
lisp/fact.lisp
View file

@ -1,6 +1,6 @@
(set! fact
(lambda (n)
"Compute the factorial of `n`, expected to be a natural number."
"Compute the factorial of `n`, expected to be an integer."
(cond ((= n 1) 1)
(t (* n (fact (- n 1)))))))

157
post-scarcity.cbp
View file

@ -1,157 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_project_file>
<FileVersion major="1" minor="6" />
<Project>
<Option title="post-scarcity" />
<Option makefile_is_custom="1" />
<Option pch_mode="2" />
<Option compiler="gcc" />
<Build>
<Target title="Debug">
<Option output="bin/Debug/post-scarcity" prefix_auto="1" extension_auto="1" />
<Option object_output="obj/Debug/" />
<Option type="1" />
<Option compiler="gcc" />
<Compiler>
<Add option="-g" />
</Compiler>
</Target>
<Target title="Release">
<Option output="bin/Release/post-scarcity" prefix_auto="1" extension_auto="1" />
<Option object_output="obj/Release/" />
<Option type="1" />
<Option compiler="gcc" />
<Compiler>
<Add option="-O2" />
</Compiler>
<Linker>
<Add option="-s" />
</Linker>
</Target>
</Build>
<Compiler>
<Add option="-Wall" />
</Compiler>
<Unit filename="Makefile" />
<Unit filename="src/arith/integer.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/arith/integer.h" />
<Unit filename="src/arith/peano.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/arith/peano.h" />
<Unit filename="src/arith/ratio.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/arith/ratio.h" />
<Unit filename="src/arith/real.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/arith/real.h" />
<Unit filename="src/authorise.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/authorise.h" />
<Unit filename="src/debug.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/debug.h" />
<Unit filename="src/init.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/io/fopen.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/io/fopen.h" />
<Unit filename="src/io/io.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/io/io.h" />
<Unit filename="src/io/print.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/io/print.h" />
<Unit filename="src/io/read.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/io/read.h" />
<Unit filename="src/memory/conspage.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/conspage.h" />
<Unit filename="src/memory/consspaceobject.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/consspaceobject.h" />
<Unit filename="src/memory/cursor.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/cursor.h" />
<Unit filename="src/memory/dump.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/dump.h" />
<Unit filename="src/memory/hashmap.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/hashmap.h" />
<Unit filename="src/memory/lookup3.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/lookup3.h" />
<Unit filename="src/memory/stack.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/stack.h" />
<Unit filename="src/memory/vectorspace.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/memory/vectorspace.h" />
<Unit filename="src/ops/equal.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/ops/equal.h" />
<Unit filename="src/ops/intern.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/ops/intern.h" />
<Unit filename="src/ops/lispops.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/ops/lispops.h" />
<Unit filename="src/ops/loop.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/ops/loop.h" />
<Unit filename="src/ops/meta.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/ops/meta.h" />
<Unit filename="src/repl.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/repl.h" />
<Unit filename="src/time/psse_time.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/time/psse_time.h" />
<Unit filename="src/utils.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="src/utils.h" />
<Unit filename="src/version.h" />
<Unit filename="utils_src/debugflags/debugflags.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="utils_src/readprintwc/readprintwc.c">
<Option compilerVar="CC" />
</Unit>
<Unit filename="utils_src/tagvalcalc/tagvalcalc.c">
<Option compilerVar="CC" />
</Unit>
<Extensions>
<lib_finder disable_auto="1" />
</Extensions>
</Project>
</CodeBlocks_project_file>

58
post-scarcity.cscope_file_list
View file

@ -1,58 +0,0 @@
"/home/simon/workspace/post-scarcity/utils_src/readprintwc/readprintwc.c"
"/home/simon/workspace/post-scarcity/src/memory/vectorspace.c"
"/home/simon/workspace/post-scarcity/src/arith/peano.c"
"/home/simon/workspace/post-scarcity/src/init.c"
"/home/simon/workspace/post-scarcity/src/utils.h"
"/home/simon/workspace/post-scarcity/src/ops/intern.h"
"/home/simon/workspace/post-scarcity/src/arith/ratio.h"
"/home/simon/workspace/post-scarcity/src/io/io.c"
"/home/simon/workspace/post-scarcity/src/memory/conspage.h"
"/home/simon/workspace/post-scarcity/src/time/psse_time.h"
"/home/simon/workspace/post-scarcity/src/memory/cursor.h"
"/home/simon/workspace/post-scarcity/src/memory/dump.h"
"/home/simon/workspace/post-scarcity/src/ops/intern.c"
"/home/simon/workspace/post-scarcity/src/memory/lookup3.c"
"/home/simon/workspace/post-scarcity/src/io/fopen.h"
"/home/simon/workspace/post-scarcity/src/version.h"
"/home/simon/workspace/post-scarcity/src/memory/consspaceobject.h"
"/home/simon/workspace/post-scarcity/src/ops/meta.h"
"/home/simon/workspace/post-scarcity/src/arith/real.c"
"/home/simon/workspace/post-scarcity/src/ops/loop.c"
"/home/simon/workspace/post-scarcity/src/arith/integer.h"
"/home/simon/workspace/post-scarcity/src/time/psse_time.c"
"/home/simon/workspace/post-scarcity/src/memory/vectorspace.h"
"/home/simon/workspace/post-scarcity/src/memory/hashmap.c"
"/home/simon/workspace/post-scarcity/src/io/read.c"
"/home/simon/workspace/post-scarcity/src/ops/lispops.h"
"/home/simon/workspace/post-scarcity/src/ops/loop.h"
"/home/simon/workspace/post-scarcity/src/memory/stack.h"
"/home/simon/workspace/post-scarcity/utils_src/tagvalcalc/tagvalcalc.c"
"/home/simon/workspace/post-scarcity/src/debug.c"
"/home/simon/workspace/post-scarcity/src/io/read.h"
"/home/simon/workspace/post-scarcity/src/ops/meta.c"
"/home/simon/workspace/post-scarcity/src/memory/dump.c"
"/home/simon/workspace/post-scarcity/src/repl.c"
"/home/simon/workspace/post-scarcity/src/io/print.c"
"/home/simon/workspace/post-scarcity/src/memory/hashmap.h"
"/home/simon/workspace/post-scarcity/src/utils.c"
"/home/simon/workspace/post-scarcity/src/io/io.h"
"/home/simon/workspace/post-scarcity/src/memory/stack.c"
"/home/simon/workspace/post-scarcity/utils_src/debugflags/debugflags.c"
"/home/simon/workspace/post-scarcity/src/memory/consspaceobject.c"
"/home/simon/workspace/post-scarcity/src/memory/conspage.c"
"/home/simon/workspace/post-scarcity/src/memory/cursor.c"
"/home/simon/workspace/post-scarcity/src/arith/ratio.c"
"/home/simon/workspace/post-scarcity/Makefile"
"/home/simon/workspace/post-scarcity/src/arith/peano.h"
"/home/simon/workspace/post-scarcity/src/memory/lookup3.h"
"/home/simon/workspace/post-scarcity/src/arith/real.h"
"/home/simon/workspace/post-scarcity/src/ops/equal.c"
"/home/simon/workspace/post-scarcity/src/ops/lispops.c"
"/home/simon/workspace/post-scarcity/src/authorise.h"
"/home/simon/workspace/post-scarcity/src/io/print.h"
"/home/simon/workspace/post-scarcity/src/authorise.c"
"/home/simon/workspace/post-scarcity/src/debug.h"
"/home/simon/workspace/post-scarcity/src/arith/integer.c"
"/home/simon/workspace/post-scarcity/src/ops/equal.h"
"/home/simon/workspace/post-scarcity/src/repl.h"
"/home/simon/workspace/post-scarcity/src/io/fopen.c"

15
post-scarcity.layout
View file

@ -1,15 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<CodeBlocks_layout_file>
<FileVersion major="1" minor="0" />
<ActiveTarget name="Debug" />
<File name="Makefile" open="1" top="0" tabpos="1" split="0" active="1" splitpos="0" zoom_1="0" zoom_2="0">
<Cursor>
<Cursor1 position="642" topLine="5" />
</Cursor>
</File>
<File name="src/arith/integer.c" open="1" top="1" tabpos="2" split="0" active="1" splitpos="0" zoom_1="0" zoom_2="0">
<Cursor>
<Cursor1 position="3454" topLine="156" />
</Cursor>
</File>
</CodeBlocks_layout_file>

88
src/arith/integer.c
View file

@ -12,7 +12,6 @@
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
/*
* wide characters
*/
@ -33,38 +32,9 @@ const char *hex_digits = "0123456789ABCDEF";
/*
* Doctrine from here on in is that ALL integers are bignums, it's just
* that integers less than 61 bits are bignums of one cell only.
* that integers less than 65 bits are bignums of one cell only.
*/
/**
* Low level integer arithmetic, do not use elsewhere.
*
* @param c a pointer to a cell, assumed to be an integer cell;
* @param op a character representing the operation: expectedto be either
* '+' or '*'; behaviour with other values is undefined.
* @param is_first_cell true if this is the first cell in a bignum
* chain, else false.
* \see multiply_integers
* \see add_integers
*/
__int128_t cell_value( struct cons_pointer c, char op, bool is_first_cell ) {
long int val = nilp( c ) ? 0 : pointer2cell( c ).payload.integer.value;
long int carry = is_first_cell ? 0 : ( INT_CELL_BASE );
__int128_t result = ( __int128_t ) integerp( c ) ?
( val == 0 ) ? carry : val : op == '*' ? 1 : 0;
debug_printf( DEBUG_ARITH,
L"cell_value: raw value is %ld, is_first_cell = %s; '%4.4s'; returning ",
val, is_first_cell ? "true" : "false",
pointer2cell( c ).tag.bytes );
debug_print_128bit( result, DEBUG_ARITH );
debug_println( DEBUG_ARITH );
return result;
}
/**
* Allocate an integer cell representing this `value` and return a cons_pointer to it.
* @param value an integer value;
@ -75,12 +45,6 @@ struct cons_pointer make_integer( int64_t value, struct cons_pointer more ) {
struct cons_pointer result = NIL;
debug_print( L"Entering make_integer\n", DEBUG_ALLOC );
if ( integerp(more) && (pointer2cell( more ).payload.integer.value < 0))
{
printf("WARNING: negative value %" PRId64 " passed as `more` to `make_integer`\n",
pointer2cell( more ).payload.integer.value);
}
if ( integerp( more ) || nilp( more ) ) {
result = allocate_cell( INTEGERTV );
struct cons_space_object *cell = &pointer2cell( result );
@ -93,12 +57,39 @@ struct cons_pointer make_integer( int64_t value, struct cons_pointer more ) {
return result;
}
/**
* Low level integer arithmetic, do not use elsewhere.
*
* @param c a pointer to a cell, assumed to be an integer cell;
* @param op a character representing the operation: expectedto be either
* '+' or '*'; behaviour with other values is undefined.
* @param is_first_cell true if this is the first cell in a bignum
* chain, else false.
* \see multiply_integers
* \see add_integers
*/
__int128_t cell_value( struct cons_pointer c, char op, bool is_first_cell ) {
long int val = nilp( c ) ? 0 : pointer2cell( c ).payload.integer.value;
long int carry = is_first_cell ? 0 : ( MAX_INTEGER + 1 );
__int128_t result = ( __int128_t ) integerp( c ) ?
( val == 0 ) ? carry : val : op == '*' ? 1 : 0;
debug_printf( DEBUG_ARITH,
L"cell_value: raw value is %ld, is_first_cell = %s; '%4.4s'; returning ",
val, is_first_cell ? "true" : "false",
pointer2cell( c ).tag.bytes );
debug_print_128bit( result, DEBUG_ARITH );
debug_println( DEBUG_ARITH );
return result;
}
/**
* Overwrite the value field of the integer indicated by `new` with
* the least significant INTEGER_BITS bits of `val`, and return the
* more significant bits (if any) right-shifted by INTEGER_BITS places.
* Destructive, primitive, do not use in any context except primitive
* operations on integers.
* the least significant 60 bits of `val`, and return the more significant
* bits (if any) right-shifted by 60 places. Destructive, primitive, do not
* use in any context except primitive operations on integers.
*
* @param val the value to represent;
* @param less_significant the less significant words of this bignum, if any,
@ -109,20 +100,21 @@ struct cons_pointer make_integer( int64_t value, struct cons_pointer more ) {
__int128_t int128_to_integer( __int128_t val,
struct cons_pointer less_significant,
struct cons_pointer new ) {
struct cons_pointer cursor = NIL;
__int128_t carry = 0;
if ( MAX_INTEGER >= val ) {
carry = 0;
} else {
carry = val % INT_CELL_BASE;
carry = val >> 60;
debug_printf( DEBUG_ARITH,
L"int128_to_integer: 64 bit overflow; setting carry to %ld\n",
( int64_t ) carry );
val /= INT_CELL_BASE;
val &= MAX_INTEGER;
}
struct cons_space_object *newc = &pointer2cell( new );
newc->payload.integer.value = (int64_t)val;
newc->payload.integer.value = val;
if ( integerp( less_significant ) ) {
struct cons_space_object *lsc = &pointer2cell( less_significant );
@ -144,7 +136,7 @@ struct cons_pointer make_integer_128( __int128_t val,
less_significant =
make_integer( ( long int ) val & MAX_INTEGER,
less_significant );
val = val * INT_CELL_BASE;
val = val >> 60;
}
} while ( nilp( result ) );
@ -298,7 +290,7 @@ struct cons_pointer multiply_integers( struct cons_pointer a,
/* if xj exceeds one digit, break it into the digit dj and
* the carry */
carry = xj >> INTEGER_BIT_SHIFT;
carry = xj >> 60;
struct cons_pointer dj = make_integer( xj & MAX_INTEGER, NIL );
/* destructively modify ri by appending dj */
@ -328,7 +320,7 @@ struct cons_pointer multiply_integers( struct cons_pointer a,
}
/**
* don't use; private to integer_to_string, and somewhat dodgy.
* don't use; private to integer_to_string, and somewaht dodgy.
*/
struct cons_pointer integer_to_string_add_digit( int digit, int digits,
struct cons_pointer tail ) {
@ -369,7 +361,7 @@ struct cons_pointer integer_to_string( struct cons_pointer int_pointer,
while ( accumulator > 0 || !nilp( next ) ) {
if ( accumulator < MAX_INTEGER && !nilp( next ) ) {
accumulator +=
( pointer2cell( next ).payload.integer.value % INT_CELL_BASE );
( pointer2cell( next ).payload.integer.value << 60 );
next = pointer2cell( next ).payload.integer.more;
}
int offset = ( int ) ( accumulator % base );

17
src/arith/peano.h
View file

@ -13,22 +13,9 @@
#define PEANO_H
/**
* The maximum value we will allow in an integer cell: one less than 2^60:
* (let ((s (make-string-output-stream)))
* (format s "0x0~XL" (- (expt 2 60) 1))
* (string-downcase (get-output-stream-string s)))
* "0x0fffffffffffffffl"
*
* So left shifting and right shifting by 60 bits is correct.
* The maximum value we will allow in an integer cell.
*/
#define MAX_INTEGER ((__int128_t)0x0fffffffffffffffL)
#define INT_CELL_BASE ((__int128_t)MAX_INTEGER + 1) // ((__int128_t)0x1000000000000000L)
/**
* @brief Number of value bits in an integer cell
*
*/
#define INTEGER_BIT_SHIFT (60)
#define MAX_INTEGER ((__int128_t)0x0fffffffffffffffL)
bool zerop( struct cons_pointer arg );

1
src/init.c
View file

@ -9,7 +9,6 @@
* Licensed under GPL version 2.0, or, at your option, any later version.
*/
#include <getopt.h>
#include <locale.h>
#include <stdbool.h>
#include <stdio.h>

14
src/io/history.c
View file

@ -1,14 +0,0 @@
/*
* history.c
*
* Maintain, and recall, a history of things which have been read from standard
* input. Necessarily the history must be stored on the user session, and not be
* global.
*
* I *think* history will be maintained as a list of forms, not of strings, so
* only forms which have successfully been read can be recalled, and forms which
* have not been completed when the history function is invoked will be lost.
*
* (c) 2025 Simon Brooke <simon@journeyman.cc>
* Licensed under GPL version 2.0, or, at your option, any later version.
*/

14
src/io/history.h
View file

@ -1,14 +0,0 @@
/*
* history.h
*
* Maintain, and recall, a history of things which have been read from standard
* input. Necessarily the history must be stored on the user session, and not be
* global.
*
* I *think* history will be maintained as a list of forms, not of strings, so
* only forms which have successfully been read can be recalled, and forms which
* have not been completed when the history function is invoked will be lost.
*
* (c) 2025 Simon Brooke <simon@journeyman.cc>
* Licensed under GPL version 2.0, or, at your option, any later version.
*/

2
src/io/print.h
View file

@ -11,8 +11,6 @@
#include <ctype.h>
#include <stdio.h>
#include "io/fopen.h"
#ifndef __print_h
#define __print_h

30
src/io/read.c
View file

@ -32,16 +32,6 @@
#include "arith/real.h"
#include "memory/vectorspace.h"
// We can't, I think, use libreadline, because we read character by character,
// not line by line, and because we use wide characters. So we're going to have
// to reimplement it. So we're going to have to maintain history of the forms
// (or strings, but I currently think forms). So we're going to have to be able
// to detact special keys, particularly, at this stage, the uparrow and down-
// arrow keys
// #include <readline/readline.h>
// #include <readline/history.h>
/*
* for the time being things which may be read are:
* * strings
@ -93,7 +83,7 @@ struct cons_pointer read_path( URL_FILE * input, wint_t initial,
prefix = c_string_to_lisp_symbol( L"oblist" );
break;
case '$':
case LSESSION:
case L'§':
prefix = c_string_to_lisp_symbol( L"session" );
break;
}
@ -255,7 +245,7 @@ struct cons_pointer read_continuation( struct stack_frame *frame,
}
break;
case '$':
case LSESSION:
case L'§':
result = read_path( input, c, NIL );
break;
default:
@ -308,9 +298,9 @@ struct cons_pointer read_number( struct stack_frame *frame,
initial );
for ( c = initial; iswdigit( c )
|| c == LPERIOD || c == LSLASH || c == LCOMMA; c = url_fgetwc( input ) ) {
|| c == L'.' || c == L'/' || c == L','; c = url_fgetwc( input ) ) {
switch ( c ) {
case LPERIOD:
case L'.':
if ( seen_period || !nilp( dividend ) ) {
return throw_exception( c_string_to_lisp_string
( L"Malformed number: too many periods" ),
@ -321,7 +311,7 @@ struct cons_pointer read_number( struct stack_frame *frame,
seen_period = true;
}
break;
case LSLASH:
case L'/':
if ( seen_period || !nilp( dividend ) ) {
return throw_exception( c_string_to_lisp_string
( L"Malformed number: dividend of rational must be integer" ),
@ -334,8 +324,8 @@ struct cons_pointer read_number( struct stack_frame *frame,
result = make_integer( 0, NIL );
}
break;
case LCOMMA:
// silently ignore comma.
case L',':
// silently ignore it.
break;
default:
result = add_integers( multiply_integers( result, base ),
@ -412,7 +402,7 @@ struct cons_pointer read_list( struct stack_frame *frame,
for ( c = url_fgetwc( input );
iswblank( c ) || iswcntrl( c ); c = url_fgetwc( input ) );
if ( c == LPERIOD ) {
if ( c == L'.' ) {
/* might be a dotted pair; indeed, if we rule out numbers with
* initial periods, it must be a dotted pair. \todo Ought to check,
* howerver, that there's only one form after the period. */
@ -443,7 +433,7 @@ struct cons_pointer read_map( struct stack_frame *frame,
make_hashmap( DFLT_HASHMAP_BUCKETS, NIL, TRUE );
wint_t c = initial;
while ( c != LCBRACE ) {
while ( c != L'}' ) {
struct cons_pointer key =
read_continuation( frame, frame_pointer, env, input, c );
@ -456,7 +446,7 @@ struct cons_pointer read_map( struct stack_frame *frame,
/* skip commaa and whitespace at this point. */
for ( c = url_fgetwc( input );
c == LCOMMA || iswblank( c ) || iswcntrl( c );
c == L',' || iswblank( c ) || iswcntrl( c );
c = url_fgetwc( input ) );
result =

9
src/io/read.h
View file

@ -13,15 +13,6 @@
#include "memory/consspaceobject.h"
/* characters (other than arabic numberals) used in number representations */
#define LCOMMA L','
#define LPERIOD L'.'
#define LSLASH L'/'
/* ... used in map representations */
#define LCBRACE L'}'
/* ... used in path representations */
#define LSESSION L'§'
/**
* read the next object on this input stream and return a cons_pointer to it.
*/

6
src/memory/conspage.c
View file

@ -238,7 +238,7 @@ struct cons_pointer allocate_cell( uint32_t tag ) {
total_cells_allocated++;
debug_printf( DEBUG_ALLOC,
L"Allocated cell of type '%4.4s' at %d, %d \n", cell->tag.bytes,
L"Allocated cell of type '%4.4s' at %d, %d \n", tag,
result.page, result.offset );
} else {
debug_printf( DEBUG_ALLOC, L"WARNING: Allocating non-free cell!" );
@ -267,6 +267,6 @@ void initialise_cons_pages( ) {
void summarise_allocation( ) {
fwprintf( stderr,
L"Allocation summary: allocated %lld; deallocated %lld; not deallocated %lld.\n",
total_cells_allocated, total_cells_freed, total_cells_allocated - total_cells_freed );
L"Allocation summary: allocated %lld; deallocated %lld.\n",
total_cells_allocated, total_cells_freed );
}

425
src/memory/consspaceobject.c
View file

@ -9,9 +9,9 @@
*/
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
/*
* wide characters
*/
@ -19,13 +19,13 @@
#include <wctype.h>
#include "authorise.h"
#include "debug.h"
#include "io/print.h"
#include "memory/conspage.h"
#include "memory/consspaceobject.h"
#include "debug.h"
#include "ops/intern.h"
#include "io/print.h"
#include "memory/stack.h"
#include "memory/vectorspace.h"
#include "ops/intern.h"
/**
* True if the value of the tag on the cell at this `pointer` is this `value`,
@ -33,22 +33,22 @@
* vectorspace object indicated by the cell is this `value`, else false.
*/
bool check_tag( struct cons_pointer pointer, uint32_t value ) {
bool result = false;
bool result = false;
struct cons_space_object cell = pointer2cell( pointer );
result = cell.tag.value == value;
struct cons_space_object cell = pointer2cell( pointer );
result = cell.tag.value == value;
if ( result == false ) {
if ( cell.tag.value == VECTORPOINTTV ) {
struct vector_space_object *vec = pointer_to_vso( pointer );
if ( result == false ) {
if ( cell.tag.value == VECTORPOINTTV ) {
struct vector_space_object *vec = pointer_to_vso( pointer );
if ( vec != NULL ) {
result = vec->header.tag.value == value;
}
if ( vec != NULL ) {
result = vec->header.tag.value == value;
}
}
}
}
return result;
return result;
}
/**
@ -56,17 +56,17 @@ bool check_tag( struct cons_pointer pointer, uint32_t value ) {
*
* You can't roll over the reference count. Once it hits the maximum
* value you cannot increment further.
*
*
* Returns the `pointer`.
*/
struct cons_pointer inc_ref( struct cons_pointer pointer ) {
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_space_object *cell = &pointer2cell( pointer );
if ( cell->count < MAXREFERENCE ) {
cell->count++;
}
if ( cell->count < MAXREFERENCE ) {
cell->count++;
}
return pointer;
return pointer;
}
/**
@ -74,46 +74,49 @@ struct cons_pointer inc_ref( struct cons_pointer pointer ) {
*
* If a count has reached MAXREFERENCE it cannot be decremented.
* If a count is decremented to zero the cell should be freed.
*
*
* Returns the `pointer`, or, if the cell has been freed, NIL.
*/
struct cons_pointer dec_ref( struct cons_pointer pointer ) {
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_space_object *cell = &pointer2cell( pointer );
if ( cell->count > 0 ) {
cell->count--;
if ( cell->count > 0 ) {
cell->count--;
if ( cell->count == 0 ) {
free_cell( pointer );
pointer = NIL;
if ( cell->count == 0 ) {
free_cell( pointer );
pointer = NIL;
}
}
}
return pointer;
return pointer;
}
/**
* Get the Lisp type of the single argument.
* @param pointer a pointer to the object whose type is requested.
* @return As a Lisp string, the tag of the object which is at that pointer.
*/
struct cons_pointer c_type( struct cons_pointer pointer ) {
struct cons_pointer result = NIL;
struct cons_space_object cell = pointer2cell( pointer );
struct cons_pointer result = NIL;
struct cons_space_object cell = pointer2cell( pointer );
if ( strncmp( (char *)&cell.tag.bytes, VECTORPOINTTAG, TAGLENGTH ) == 0 ) {
struct vector_space_object *vec = pointer_to_vso( pointer );
if ( strncmp( ( char * ) &cell.tag.bytes, VECTORPOINTTAG, TAGLENGTH ) ==
0 ) {
struct vector_space_object *vec = pointer_to_vso( pointer );
for ( int i = TAGLENGTH - 1; i >= 0; i-- ) {
result = make_string( (wchar_t)vec->header.tag.bytes[i], result );
for ( int i = TAGLENGTH - 1; i >= 0; i-- ) {
result =
make_string( ( wchar_t ) vec->header.tag.bytes[i], result );
}
} else {
for ( int i = TAGLENGTH - 1; i >= 0; i-- ) {
result = make_string( ( wchar_t ) cell.tag.bytes[i], result );
}
}
} else {
for ( int i = TAGLENGTH - 1; i >= 0; i-- ) {
result = make_string( (wchar_t)cell.tag.bytes[i], result );
}
}
return result;
return result;
}
/**
@ -121,13 +124,13 @@ struct cons_pointer c_type( struct cons_pointer pointer ) {
* authorised to read it, does not error but returns nil.
*/
struct cons_pointer c_car( struct cons_pointer arg ) {
struct cons_pointer result = NIL;
struct cons_pointer result = NIL;
if ( truep( authorised( arg, NIL ) ) && consp( arg ) ) {
result = pointer2cell( arg ).payload.cons.car;
}
if ( truep( authorised( arg, NIL ) ) && consp( arg ) ) {
result = pointer2cell( arg ).payload.cons.car;
}
return result;
return result;
}
/**
@ -135,98 +138,96 @@ struct cons_pointer c_car( struct cons_pointer arg ) {
* not authorised to read it,does not error but returns nil.
*/
struct cons_pointer c_cdr( struct cons_pointer arg ) {
struct cons_pointer result = NIL;
struct cons_pointer result = NIL;
if ( truep( authorised( arg, NIL ) ) ) {
struct cons_space_object *cell = &pointer2cell( arg );
if ( truep( authorised( arg, NIL ) ) ) {
struct cons_space_object *cell = &pointer2cell( arg );
switch ( cell->tag.value ) {
case CONSTV:
result = cell->payload.cons.cdr;
break;
case KEYTV:
case STRINGTV:
case SYMBOLTV:
result = cell->payload.string.cdr;
break;
switch ( cell->tag.value ) {
case CONSTV:
result = cell->payload.cons.cdr;
break;
case KEYTV:
case STRINGTV:
case SYMBOLTV:
result = cell->payload.string.cdr;
break;
}
}
}
return result;
return result;
}
/**
* Implementation of `length` in C. If arg is not a cons, does not error but
* returns 0.
* Implementation of `length` in C. If arg is not a cons, does not error but returns 0.
*/
int c_length( struct cons_pointer arg ) {
int result = 0;
int result = 0;
for ( struct cons_pointer c = arg; !nilp( c ); c = c_cdr( c ) ) {
result++;
}
for ( struct cons_pointer c = arg; !nilp( c ); c = c_cdr( c ) ) {
result++;
}
return result;
return result;
}
/**
* Construct a cons cell from this pair of pointers.
*/
struct cons_pointer make_cons( struct cons_pointer car,
struct cons_pointer cdr ) {
struct cons_pointer pointer = NIL;
struct cons_pointer pointer = NIL;
pointer = allocate_cell( CONSTV );
pointer = allocate_cell( CONSTV );
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( car );
inc_ref( cdr );
cell->payload.cons.car = car;
cell->payload.cons.cdr = cdr;
inc_ref( car );
inc_ref( cdr );
cell->payload.cons.car = car;
cell->payload.cons.cdr = cdr;
return pointer;
return pointer;
}
/**
* Construct an exception cell.
* @param message should be a lisp string describing the problem, but actually
* any cons pointer will do;
* @param frame_pointer should be the pointer to the frame in which the
* exception occurred.
* @param message should be a lisp string describing the problem, but actually any cons pointer will do;
* @param frame_pointer should be the pointer to the frame in which the exception occurred.
*/
struct cons_pointer make_exception( struct cons_pointer message,
struct cons_pointer frame_pointer ) {
struct cons_pointer result = NIL;
struct cons_pointer pointer = allocate_cell( EXCEPTIONTV );
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_pointer result = NIL;
struct cons_pointer pointer = allocate_cell( EXCEPTIONTV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( message );
inc_ref( frame_pointer );
cell->payload.exception.payload = message;
cell->payload.exception.frame = frame_pointer;
inc_ref( message );
inc_ref( frame_pointer );
cell->payload.exception.payload = message;
cell->payload.exception.frame = frame_pointer;
result = pointer;
result = pointer;
return result;
return result;
}
/**
* Construct a cell which points to an executable Lisp function.
*/
struct cons_pointer make_function(
struct cons_pointer meta,
struct cons_pointer ( *executable )( struct stack_frame *,
struct cons_pointer,
struct cons_pointer ) ) {
struct cons_pointer pointer = allocate_cell( FUNCTIONTV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( meta );
struct cons_pointer
make_function( struct cons_pointer meta, struct cons_pointer ( *executable )
( struct stack_frame *,
struct cons_pointer, struct cons_pointer ) ) {
struct cons_pointer pointer = allocate_cell( FUNCTIONTV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( meta );
cell->payload.function.meta = meta;
cell->payload.function.executable = executable;
cell->payload.function.meta = meta;
cell->payload.function.executable = executable;
return pointer;
return pointer;
}
/**
@ -234,18 +235,17 @@ struct cons_pointer make_function(
*/
struct cons_pointer make_lambda( struct cons_pointer args,
struct cons_pointer body ) {
struct cons_pointer pointer = allocate_cell( LAMBDATV );
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_pointer pointer = allocate_cell( LAMBDATV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( pointer ); /* this is a hack; I don't know why it's necessary to do
this, but if I don't the cell gets freed */
inc_ref( pointer ); /* this is a hack; I don't know why it's necessary to do this, but if I don't the cell gets freed */
inc_ref( args );
inc_ref( body );
cell->payload.lambda.args = args;
cell->payload.lambda.body = body;
inc_ref( args );
inc_ref( body );
cell->payload.lambda.args = args;
cell->payload.lambda.body = body;
return pointer;
return pointer;
}
/**
@ -254,48 +254,48 @@ struct cons_pointer make_lambda( struct cons_pointer args,
*/
struct cons_pointer make_nlambda( struct cons_pointer args,
struct cons_pointer body ) {
struct cons_pointer pointer = allocate_cell( NLAMBDATV );
struct cons_pointer pointer = allocate_cell( NLAMBDATV );
inc_ref( pointer ); /* this is a hack; I don't know why it's necessary to do
this, but if I don't the cell gets freed */
inc_ref( pointer ); /* this is a hack; I don't know why it's necessary to do this, but if I don't the cell gets freed */
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( args );
inc_ref( body );
cell->payload.lambda.args = args;
cell->payload.lambda.body = body;
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( args );
inc_ref( body );
cell->payload.lambda.args = args;
cell->payload.lambda.body = body;
return pointer;
return pointer;
}
/**
* Return a hash value for this string like thing.
*
*
* What's important here is that two strings with the same characters in the
* same order should have the same hash value, even if one was created using
* `"foobar"` and the other by `(append "foo" "bar")`. I *think* this function
* has that property. I doubt that it's the most efficient hash function to
* `"foobar"` and the other by `(append "foo" "bar")`. I *think* this function
* has that property. I doubt that it's the most efficient hash function to
* have that property.
*
*
* returns 0 for things which are not string like.
*/
uint32_t calculate_hash( wint_t c, struct cons_pointer ptr ) {
struct cons_space_object *cell = &pointer2cell( ptr );
uint32_t result = 0;
struct cons_space_object *cell = &pointer2cell( ptr );
uint32_t result = 0;
switch ( cell->tag.value ) {
case KEYTV:
case STRINGTV:
case SYMBOLTV:
if ( nilp( cell->payload.string.cdr ) ) {
result = (uint32_t)c;
} else {
result = ( (uint32_t)c * cell->payload.string.hash ) & 0xffffffff;
}
break;
}
switch ( cell->tag.value ) {
case KEYTV:
case STRINGTV:
case SYMBOLTV:
if ( nilp( cell->payload.string.cdr ) ) {
result = ( uint32_t ) c;
} else {
result = ( ( uint32_t ) c *
cell->payload.string.hash ) & 0xffffffff;
}
break;
}
return result;
return result;
}
/**
@ -304,31 +304,31 @@ uint32_t calculate_hash( wint_t c, struct cons_pointer ptr ) {
* has one character and a pointer to the next; in the last cell the
* pointer to next is NIL.
*/
struct cons_pointer make_string_like_thing( wint_t c, struct cons_pointer tail,
uint32_t tag ) {
struct cons_pointer pointer = NIL;
struct cons_pointer
make_string_like_thing( wint_t c, struct cons_pointer tail, uint32_t tag ) {
struct cons_pointer pointer = NIL;
if ( check_tag( tail, tag ) || check_tag( tail, NILTV ) ) {
pointer = allocate_cell( tag );
struct cons_space_object *cell = &pointer2cell( pointer );
if ( check_tag( tail, tag ) || check_tag( tail, NILTV ) ) {
pointer = allocate_cell( tag );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( tail );
cell->payload.string.character = c;
cell->payload.string.cdr.page = tail.page;
/* \todo There's a problem here. Sometimes the offsets on
* strings are quite massively off. Fix is probably
* cell->payload.string.cdr = tail */
cell->payload.string.cdr.offset = tail.offset;
inc_ref( tail );
cell->payload.string.character = c;
cell->payload.string.cdr.page = tail.page;
/* \todo There's a problem here. Sometimes the offsets on
* strings are quite massively off. Fix is probably
* cell->payload.string.cdr = tail */
cell->payload.string.cdr.offset = tail.offset;
cell->payload.string.hash = calculate_hash( c, tail );
} else {
// \todo should throw an exception!
debug_printf( DEBUG_ALLOC,
L"Warning: only NIL and %4.4s can be prepended to %4.4s\n",
tag, tag );
}
cell->payload.string.hash = calculate_hash( c, tail );
} else {
// \todo should throw an exception!
debug_printf( DEBUG_ALLOC,
L"Warning: only NIL and %4.4s can be prepended to %4.4s\n",
tag, tag );
}
return pointer;
return pointer;
}
/**
@ -340,7 +340,7 @@ struct cons_pointer make_string_like_thing( wint_t c, struct cons_pointer tail,
* @param tail the string which is being built.
*/
struct cons_pointer make_string( wint_t c, struct cons_pointer tail ) {
return make_string_like_thing( c, tail, STRINGTV );
return make_string_like_thing( c, tail, STRINGTV );
}
/**
@ -353,45 +353,36 @@ struct cons_pointer make_string( wint_t c, struct cons_pointer tail ) {
*/
struct cons_pointer make_symbol_or_key( wint_t c, struct cons_pointer tail,
uint32_t tag ) {
struct cons_pointer result;
if ( tag == SYMBOLTV || tag == KEYTV ) {
result = make_string_like_thing( c, tail, tag );
struct cons_pointer result = make_string_like_thing( c, tail, tag );
if ( tag == KEYTV ) {
struct cons_pointer r = internedp( result, oblist );
struct cons_pointer r = internedp( result, oblist );
if ( nilp( r ) ) {
intern( result, oblist );
} else {
result = r;
}
if ( nilp( r ) ) {
intern( result, oblist );
} else {
result = r;
}
}
} else {
result = make_exception(
c_string_to_lisp_string( L"Unexpected tag when making symbol or key." ),
NIL);
}
return result;
return result;
}
/**
* Construct a cell which points to an executable Lisp special form.
*/
struct cons_pointer make_special(
struct cons_pointer meta,
struct cons_pointer ( *executable )( struct stack_frame *frame,
struct cons_pointer,
struct cons_pointer env ) ) {
struct cons_pointer pointer = allocate_cell( SPECIALTV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( meta );
struct cons_pointer
make_special( struct cons_pointer meta, struct cons_pointer ( *executable )
( struct stack_frame * frame,
struct cons_pointer, struct cons_pointer env ) ) {
struct cons_pointer pointer = allocate_cell( SPECIALTV );
struct cons_space_object *cell = &pointer2cell( pointer );
inc_ref( meta );
cell->payload.special.meta = meta;
cell->payload.special.executable = executable;
cell->payload.special.meta = meta;
cell->payload.special.executable = executable;
return pointer;
return pointer;
}
/**
@ -400,15 +391,15 @@ struct cons_pointer make_special(
* @param metadata a pointer to an associaton containing metadata on the stream.
* @return a pointer to the new read stream.
*/
struct cons_pointer make_read_stream( URL_FILE *input,
struct cons_pointer make_read_stream( URL_FILE * input,
struct cons_pointer metadata ) {
struct cons_pointer pointer = allocate_cell( READTV );
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_pointer pointer = allocate_cell( READTV );
struct cons_space_object *cell = &pointer2cell( pointer );
cell->payload.stream.stream = input;
cell->payload.stream.meta = metadata;
cell->payload.stream.stream = input;
cell->payload.stream.meta = metadata;
return pointer;
return pointer;
}
/**
@ -417,59 +408,59 @@ struct cons_pointer make_read_stream( URL_FILE *input,
* @param metadata a pointer to an associaton containing metadata on the stream.
* @return a pointer to the new read stream.
*/
struct cons_pointer make_write_stream( URL_FILE *output,
struct cons_pointer make_write_stream( URL_FILE * output,
struct cons_pointer metadata ) {
struct cons_pointer pointer = allocate_cell( WRITETV );
struct cons_space_object *cell = &pointer2cell( pointer );
struct cons_pointer pointer = allocate_cell( WRITETV );
struct cons_space_object *cell = &pointer2cell( pointer );
cell->payload.stream.stream = output;
cell->payload.stream.meta = metadata;
cell->payload.stream.stream = output;
cell->payload.stream.meta = metadata;
return pointer;
return pointer;
}
/**
* Return a lisp keyword representation of this wide character string. In
* keywords, I am accepting only lower case characters and numbers.
* Return a lisp keyword representation of this wide character string. In keywords,
* I am accepting only lower case characters and numbers.
*/
struct cons_pointer c_string_to_lisp_keyword( wchar_t *symbol ) {
struct cons_pointer result = NIL;
struct cons_pointer result = NIL;
for ( int i = wcslen( symbol ) - 1; i >= 0; i-- ) {
wchar_t c = towlower( symbol[i] );
for ( int i = wcslen( symbol ) - 1; i >= 0; i-- ) {
wchar_t c = towlower( symbol[i] );
if ( iswalnum( c ) || c == L'-' ) {
result = make_keyword( c, result );
if ( iswalnum( c ) || c == L'-' ) {
result = make_keyword( c, result );
}
}
}
return result;
return result;
}
/**
* Return a lisp string representation of this wide character string.
*/
struct cons_pointer c_string_to_lisp_string( wchar_t *string ) {
struct cons_pointer result = NIL;
struct cons_pointer result = NIL;
for ( int i = wcslen( string ) - 1; i >= 0; i-- ) {
if ( iswprint( string[i] ) && string[i] != '"' ) {
result = make_string( string[i], result );
for ( int i = wcslen( string ) - 1; i >= 0; i-- ) {
if ( iswprint( string[i] ) && string[i] != '"' ) {
result = make_string( string[i], result );
}
}
}
return result;
return result;
}
/**
* Return a lisp symbol representation of this wide character string.
*/
struct cons_pointer c_string_to_lisp_symbol( wchar_t *symbol ) {
struct cons_pointer result = NIL;
struct cons_pointer result = NIL;
for ( int i = wcslen( symbol ); i > 0; i-- ) {
result = make_symbol( symbol[i - 1], result );
}
for ( int i = wcslen( symbol ); i > 0; i-- ) {
result = make_symbol( symbol[i - 1], result );
}
return result;
return result;
}

28
src/memory/consspaceobject.h
View file

@ -121,30 +121,6 @@
*/
#define LOOPTV 1347374924
/**
* @brief Tag for a lazy cons cell.
*
* A lazy cons cell is like a cons cell, but lazy.
*
*/
#define LAZYCONSTAG "LZYC"
/**
* @brief Tag for a lazy string cell.
*
* A lazy string cell is like a string cell, but lazy.
*
*/
#define LAZYSTRTAG "LZYS"
/**
* @brief Tag for a lazy worker cell.
*
* A lazy
*
*/
#define LAZYWRKRTAG "WRKR"
/**
* The special cons cell at address {0,0} whose car and cdr both point to
* itself.
@ -502,13 +478,11 @@ struct free_payload {
* exceeds 60 bits, the least significant 60 bits are stored in the first cell
* in the chain, the next 60 in the next cell, and so on. Only the value of the
* first cell in any chain should be negative.
*
* \todo Why is this 60, and not 64 bits?
*/
struct integer_payload {
/** the value of the payload (i.e. 60 bits) of this cell. */
int64_t value;
/** the next (more significant) cell in the chain, or `NIL` if there are no
/** the next (more significant) cell in the chain, ir `NIL` if there are no
* more. */
struct cons_pointer more;
};

24
src/memory/hashmap.c
View file

@ -155,16 +155,7 @@ struct cons_pointer lisp_make_hashmap( struct stack_frame *frame,
}
}
if ( frame->args > 1 ) {
if ( functionp( frame->arg[1])) {
hash_fn = frame->arg[1];
} else if ( nilp(frame->arg[1])){
/* that's allowed */
} else {
result =
make_exception( c_string_to_lisp_string
( L"Second arg to `hashmap`, if passed, must "
L"be a function or `nil`.`" ), NIL );
}
}
if ( nilp( result ) ) {
@ -198,23 +189,26 @@ struct cons_pointer lisp_make_hashmap( struct stack_frame *frame,
return result;
}
/**
* If this `ptr` is a pointer to a hashmap, return a new identical hashmap;
* else return an exception.
* else return `NIL`. TODO: should return an exception if ptr is not a
* readable hashmap.
*/
struct cons_pointer clone_hashmap( struct cons_pointer ptr ) {
struct cons_pointer result = NIL;
if ( truep( authorised( ptr, NIL ) ) ) {
if ( hashmapp( ptr ) ) {
struct vector_space_object const *from = pointer_to_vso( ptr );
struct vector_space_object *from = pointer_to_vso( ptr );
if ( from != NULL ) {
struct hashmap_payload from_pl = from->payload.hashmap;
result =
make_hashmap( from_pl.n_buckets, from_pl.hash_fn,
from_pl.write_acl );
struct vector_space_object const *to = pointer_to_vso( result );
struct vector_space_object *to = pointer_to_vso( result );
struct hashmap_payload to_pl = to->payload.hashmap;
for ( int i = 0; i < to_pl.n_buckets; i++ ) {
@ -223,12 +217,8 @@ struct cons_pointer clone_hashmap( struct cons_pointer ptr ) {
}
}
}
} else {
result =
make_exception( c_string_to_lisp_string
( L"Arg to `clone_hashmap` must "
L"be a readable hashmap.`" ), NIL );
}
// TODO: else exception?
return result;
}

7
src/memory/vectorspace.c
View file

@ -84,11 +84,10 @@ struct cons_pointer make_vso( uint32_t tag, uint64_t payload_size ) {
if ( vso != NULL ) {
memset( vso, 0, padded );
vso->header.tag.value = tag;
debug_printf( DEBUG_ALLOC,
L"make_vso: written tag '%4.4s' into vso at %p\n",
vso->header.tag.bytes, vso );
L"make_vso: about to write tag '%4.4s' into vso at %p\n",
tag, vso );
vso->header.tag.value = tag;
result = make_vec_pointer( vso, tag );
debug_dump_object( result, DEBUG_ALLOC );
vso->header.vecp = result;

3
src/ops/lispops.c
View file

@ -681,8 +681,6 @@ bool end_of_stringp( struct cons_pointer arg ) {
* returns a cell constructed from a and b. If a is of type string but its
* cdr is nill, and b is of type string, then returns a new string cell;
* otherwise returns a new cons cell.
*
* Thus: `(cons "a" "bcd") -> "abcd"`, but `(cons "ab" "cd") -> ("ab" . "cd")`
*
* * (cons a b)
*
@ -702,6 +700,7 @@ lisp_cons( struct stack_frame *frame, struct cons_pointer frame_pointer,
return NIL;
} else if ( stringp( car ) && stringp( cdr ) &&
end_of_stringp( c_cdr( car ) ) ) {
// \todo check that car is of length 1
result =
make_string( pointer2cell( car ).payload.string.character, cdr );
} else {

2
src/version.h
View file

@ -8,4 +8,4 @@
* Licensed under GPL version 2.0, or, at your option, any later version.
*/
#define VERSION "0.0.6-SNAPSHOT"
#define VERSION "0.0.5"

62
state-of-play.md
View file

@ -1,62 +0,0 @@
# State of Play
## 20250704
Right, I'm getting second and subsequent integer cells with negative values, which should not happen. This is probably the cause of (at least some of) the bignum problems. I need to find out why. This is (probably) fixable.
```lisp
:: (inspect 10000000000000000000)
INTR (1381256777) at page 3, offset 873 count 2
Integer cell: value 776627963145224192, count 2
BIGNUM! More at:
INTR (1381256777) at page 3, offset 872 count 1
Integer cell: value -8, count 1
```
Also, `print` is printing bignums wrong on ploughwright, but less wrong on mason, which implies a code difference. Investigate.
## 20250314
Thinking further about this, I think at least part of the problem is that I'm storing bignums as cons-space objects, which means that the integer representation I can store has to fit into the size of a cons pointer, which is 64 bits. Which means that to store integers larger than 64 bits I need chains of these objects.
If I stored bignums in vector space, this problem would go away (especially as I have not implemented vector space yet).
However, having bignums in vector space would cause a churn of non-standard-sized objects in vector space, which would mean much more frequent garbage collection, which has to be mark-and-sweep because unequal-sized objects, otherwise you get heap fragmentation.
So maybe I just have to put more work into debugging my cons-space bignums.
Bother, bother.
There are no perfect solutions.
However however, it's only the node that's short on vector space which has to pause to do a mark and sweep. It doesn't interrupt any other node, because their reference to the object will remain the same, even if it is the 'home node' of the object which is sweeping. So all the node has to do is set its busy flag, do GC, and clear its busy flag, The rest of the system can just be carrying on as normal.
So... maybe mark and sweep isn't the big deal I think it is?
## 20250313
OK, the 60 bit integer cell happens in `int128_to_integer` in `arith/integer.c`. It seems to be being done consistently; but there is no obvious reason. `MAX_INTEGER` is defined in `arith/peano.h`. I've changed both to use 63 bits, and this makes no change to the number of unit tests that fail.
With this change, `(fact 21)`, which was previously printing nothing, now prints a value, `11,891,611,015,076,642,816`. However, this value is definitively wrong, should be `51,090,942,171,709,440,000`. But, I hadn't fixed the shift in `integer_to_string`; have now... still no change in number of failed tests...
But `(fact 21)` gives a different wrong value, `4,974,081,987,435,560,960`. Factorial values returned by `fact` are correct (agree with SBCL running the same code) up to `(fact 20)`, with both 60 bit integer cells and 63 bit integer cells giving correct values.
Uhhhmmm... but I'd missed two other places where I'd had the number of significant bits as a numeric literal. Fixed those and now `(fact 21)` does not return a printable answer at all, although the internal representation is definitely wrong. So we may be seeing why I chose 60 bits.
Bother.
## 20250312
Printing of bignums definitely doesn't work; I'm not persuaded that reading of bignums works right either, and there are probably problems with bignum arithmetic too.
The internal memory representation of a number rolls over from one cell to two cells at 1152921504606846976, and I'm not at all certain why it does because this is neither 2<sup>63</sup> nor 2<sup>64</sup>.
| | | |
| -------------- | -------------------- | ---- |
| 2<sup>62</sup> | 4611686018427387904 | |
| 2<sup>63</sup> | 9223372036854775808 | |
| 2<sup>64</sup> | 18446744073709551616 | |
| Mystery number | 1152921504606846976 | |
In fact, our mystery number turns out (by inspection) to be 2<sup>60</sup>. But **why**?

45
unit-tests/bignum-add.sh
View file

@ -129,51 +129,6 @@ else
fi
#####################################################################
# add two small bignums to produce a bigger bignum
a=1152921504606846977
c=`echo "$a + $a" | bc`
echo -n "adding $a to $a: "
expected='t'
output=`echo "(= (+ $a $b) $c)" | target/psse -v 2 2>psse.log`
actual=`echo $output |\
tail -1 |\
sed 's/\,//g'`
if [ "${expected}" = "${actual}" ]
then
echo "OK"
else
echo "Fail: expected '${expected}', got '${actual}'"
exit 1
fi
#####################################################################
# add five small bignums to produce a bigger bignum
a=1152921504606846977
c=`echo "$a * 5" | bc`
echo -n "adding $a, $a $a, $a, $a: "
expected='t'
output=`echo "(= (+ $a $a $a $a $a) $c)" | target/psse -v 2 2>psse.log`
actual=`echo $output |\
tail -1 |\
sed 's/\,//g'`
if [ "${expected}" = "${actual}" ]
then
echo "OK"
else
echo "Fail: expected '${expected}', got '${actual}'"
exit 1
fi
#####################################################################
# add two bignums to produce a bignum
a=10000000000000000000

14
unit-tests/memory.sh
View file

@ -1,14 +0,0 @@
#!/bin/bash
actual=`echo "" | target/psse 2>&1 | tail -2`
alloc=`echo $actual | sed 's/[[:punct:]]/ /g' | awk '{print $4}'`
dealloc=`echo $actual | sed 's/[[:punct:]]/ /g' | awk '{print $6}'`
if [ "${alloc}" = "${dealloc}" ]
then
echo "OK"
else
echo "Fail: expected '${alloc}', got '${dealloc}'"
exit 1
fi