diff --git a/.gitignore b/.gitignore
index 99b78ef..e560793 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,6 +9,7 @@ pom.xml.asc
/checkouts/
/resources/public/content/.git
/resources/public/vendor
+/bower_components/
.lein-deps-sum
.lein-repl-history
.lein-plugins/
diff --git a/doc/specification/competitors.md b/doc/specification/competitors.md
new file mode 100644
index 0000000..8fa7dd0
--- /dev/null
+++ b/doc/specification/competitors.md
@@ -0,0 +1,23 @@
+# Competitor Analysis
+
+Obviously **You Yes Yet?** is my baby; I've put a lot of thought into it. At the time I started working on it I wasn't aware of any open source competitors; I did to a web search, and I emailed the Bernie Sanders campaign to see whether their widely admired tools were open source. I didn't find anything.
+
+However, I've just been pointed to [Vote Leave](http://www.voteleavetakecontrol.org/)'s [Vics](https://dominiccummings.wordpress.com/2016/10/29/on-the-referendum-20-the-campaign-physics-and-data-science-vote-leaves-voter-intention-collection-system-vics-now-available-for-all/) tool, and there may well be others.
+
+There is no room here for ego. What matters is that the Yes campaign gets the best available tool for the job. So it's important to do competitor analysis, and not to invest too much work into **You Yes Yet?** unless there's a realistic possibility of producing a tool which is better than any of the available alternatives. But it's also the case that by studying competitors we may find ways to improve the design of **You Yes Yet?**.
+
+## Vics
+
+Vics, the **Voter Intention Collection System**, is reputed to have been a significant factor in the successful campaign by Vote Leave to take Britain out of the EU. It has been [released](https://github.com/celestial-winter/vics) as open source under MIT licence, so it is unambiguously available for us to use.
+
+The architecture comprises a single-page app built using Angular talking to a server built in Java using the Spring framework. The database engine used is Postgres. [Jedis](https://github.com/xetorthio/jedis), a Java port of [Redis](https://redis.io/), is used as an in-memory data cache, server side.
+
+### Download and initial build
+
+I checked out the source from the GitHub repository, and following the instructions in the README created the database and ran a maven install process. Unfortunately, run as a normal user, when this process goes into its test sequence many tests fail unable to contact Jedis. I find it slightly worrying to run such a large and complex build as *root*, but as *root* it gets substantially further. The build still doesn't complete but it seems that it is closer to completion.
+
+The ironic point is that it fails because it depends on the JavaScript package manager [bower](https://bower.io/), and bower (very sensibly) refuses to run as *root*. I therefore made a small modification to the build script to allow it to run *bower* as root, but unfortunately that didn't solve the build problem; the jedis service was still not found where it was expected.
+
+This is difficult to diagnose; the exception is so deeply nested in framework code that no code from the actual Vics application appears on the stack dump, which makes it very hard to know where to start in debugging.
+
+So for tonight I've failed. I shall try again.
diff --git a/doc/specification/scaling.md b/doc/specification/scaling.md
index 5a8de4d..bfc16de 100644
--- a/doc/specification/scaling.md
+++ b/doc/specification/scaling.md
@@ -42,6 +42,22 @@ This means that the maximum number of transactions per second across Scotland is
## Spreading the load
+### Caching and memoizing
+
+People typically go out canvassing in teams; each member of the team will need the same elector data.
+
+Glasgow has a population density of about 3,260 per Km^2; that means each half kilometer square has a maximum population of not much more than 1,000. Downloading 1,000 elector records at startup time is not infeasible. If we normalise data requests to a 100 metre square grid and serve records in 500 metre square chunks, all the members of the same team will request the same chunk of data. Also, elector data is not volatile. Therefore it makes sense to memoize requests for elector data. The app should only request fresh elector data when the device moves within 100 metres of the edge of the current 500 metre cell.
+
+Intention data is volatile: we'll want to update canvassers with fresh intention data frequently, because the other members of their team will be recording intention data as they work, and it's by seeing that intention data that the canvassers know which doors are still unchapped. So we don't want to cache intention data for very long. But nevertheless it still makes sense to deliver it in normalised 500 metre square chunks, because that means we can temporarily cache it server side and do not actually have to hit the database with many requests for the same data.
+
+Finally, issue data is not volatile over the course of a canvassing session, although it may change over a period of days. So issue data - all the current issues - should be fetched once at app startup time, and not periodically refreshed during a canvassing session. Also, of course, every canvasser will require exactly the same issue data (unless we start thinking of local or regional issues...?), so it absolutely makes sense to memoise requests for issue data.
+
+All this normalisation and memoisation reduces the number of read requests on the database.
+
+Note that [clojure.core.memoize](https://github.com/clojure/core.memoize) provides us with functions to create both size-limited, least-recently-used caches and duration limited, time-to-live caches.
+
+At 56 degrees north there are 111,341 metres per degree of latitude, 62,392 metres per degree of longitude. So a 100 metre box is about 0.0016 degrees east-west and .0009 degrees north-south. If we simplify that slightly (and we don't need square boxes, we need units of area covering a group of people working together) then we can take .001 of a degree in either direction which is computationally cheap.
+
### Geographic sharding
Volunteers canvassing simultaneously in the same street or the same locality need to see in near real time which dwellings have been canvassed by other volunteers, otherwise we'll get the same households canvassed repeatedly, which wastes volunteer time and annoys voters. So they all need to be sending updates to, and receiving updates from, the same server. But volunteers canvassing in Aberdeen don't need to see in near real time what is happening in Edinburgh.
@@ -56,6 +72,28 @@ The geographic sharding strategy is scalable. We could start with a single serve
But having considerable numbers of database servers will have cost implications.
+### Geographic sharding by DNS
+
+When I first thought of geographic sharding, I intended sharding by electoral district, but actually that makes no sense because electoral districts are complex polygons, which makes point-within-polygon computationally expensive. 4 degrees west falls just west of Stirling, and divides the country in half north-south. 56 degrees north runs north of Edinburgh and Glasgow, but just south of Falkirk. It divides the country in half east to west. Very few towns (and no large towns) straddle either line. Thus we can divide Scotland neatly into four, and it is computationally extremely cheap to compute which shard each data item should be despatched to.
+
+We can then set up in DNS four addresses:
+
+ +----------------------+-----------+-----------+
+ | Address | longitude | latitude |
+ +----------------------+-----------+-----------+
+ | nw.data.yyy.scot | < -4 | >= 56 |
+ +----------------------+-----------+-----------+
+ | ne.data.yyy.scot | >= -4 | >= 56 |
+ +----------------------+-----------+-----------+
+ | sw.data.yyy.scot | < -4 | < 56 |
+ +----------------------+-----------+-----------+
+ | se.data.yyy.scot | >= -4 | < 56 |
+ +----------------------+-----------+-----------+
+
+giving us an incredibly simple dispatch table. Furthermore, initially all four addresses can point to the same server. This is an incredibly simple scheme, and I'm confident it's good enough.
+
+Data that's inserted from the canvassing app - that is to say, voter intention data and followup request data - should have an additional field 'shard' (char(2)) which should hold the digraph representing the shard to which it was dispatched, and that field should form part of the primary key, allowing the data from all servers to be integrated. Data that isn't from the canvassing app should probably be directed to the 'nw' shard (which will be lightest loaded), or to a separate master server, and then all servers should be synced overnight.
+
### Read servers and write servers
It's a common practice in architecting busy web systems to have one master database server to which all write operations are directed, surrounded by a ring of slave databases which replicate from the master and serve all read requests. This works because for the majority of web systems there are many more reads than writes.
@@ -76,15 +114,13 @@ From the above I think the scaling problem should be addressed as follows:
4. When the initial cluster of three database servers becomes overloaded, shard into two identical groups ('east' and 'west');
5. When any shard becomes overloaded, split it into two further shards.
-If we have prepared for sharding, all that is required is to duplicate the database and then set geographic polygons to address database requests from clients within a given polygon to the database server for that polygon.
-
-This means that essentially we should set up one polygon for each electoral district from the launch of the app, but initially requests from all of these polygons should be directed to the same database server group. As shards are added, the map of polygons to database server groups should be updated.
+If we have prepared for sharding, all that is required is to duplicate the database.
Obviously, once we have split the database into multiple shards, there is a task to integrate the data from the multiple shards in order to create an 'across Scotland' overview of the canvas data; however, again if we have prepared for it in advance, merging the databases should not be difficult, and can be done either in the wee sma' oors or alternatively during the working day, as the system will be relatively lighty loaded during these periods.
## Preparing for sharding
-We should prepare a Docker image for the app server and a Docker image for the database server. We should prepare, as part of the app (i.e. not in the database but as a Clojure or Json data file) a datastructure which maps polygons representing each of Scotland's electoral districts to database URLs. For security reasons this datastructure should live server-side and should not be part of the single-page app sent to the client.
+We should prepare a Docker image for the app server and an image or setup script for the database server.
## Further reading on optimising Postgres performance
diff --git a/env/dev/clj/user.clj b/env/dev/clj/user.clj
index a8adafd..97cb362 100644
--- a/env/dev/clj/user.clj
+++ b/env/dev/clj/user.clj
@@ -13,4 +13,23 @@
(stop)
(start))
+;; Roughly working under Tomcat.
+;; Database setup using JNDI: see http://www.luminusweb.net/docs/deployment.md#deploying_to_tomcat
+;; Note that this duplicates configuration in profiles.clj; one of these is wrong (and neither
+;; should be in the Git repository but this is for now!)
+(System/setProperty "java.naming.factory.initial"
+ "org.apache.naming.java.javaURLContextFactory")
+(System/setProperty "java.naming.factory.url.pkgs"
+ "org.apache.naming")
+(doto (new javax.naming.InitialContext)
+ (.createSubcontext "java:")
+ (.createSubcontext "java:comp")
+ (.createSubcontext "java:comp/env")
+ (.createSubcontext "java:comp/env/jdbc")
+ (.bind "java:comp/env/jdbc/testdb"
+ (doto (org.postgresql.ds.PGSimpleDataSource.)
+ (.setServerName "localhost")
+ (.setDatabaseName "youyesyet_dev")
+ (.setUser "youyesyet")
+ (.setPassword "thisisnotsecure"))))
diff --git a/favicon.ico b/favicon.ico
new file mode 100644
index 0000000..5df78a1
Binary files /dev/null and b/favicon.ico differ
diff --git a/favicon.xcf b/favicon.xcf
new file mode 100644
index 0000000..643b2f0
Binary files /dev/null and b/favicon.xcf differ
diff --git a/followup.cljs b/followup.cljs
new file mode 100644
index 0000000..836d622
--- /dev/null
+++ b/followup.cljs
@@ -0,0 +1,72 @@
+(ns youyesyet.views.followup
+ (:require [reagent.core :as r]
+ [re-frame.core :refer [reg-sub subscribe]]
+;; [re-frame-forms.core :as form]
+;; [re-frame-forms.input :as input]
+;; [re-com.core :refer [h-box v-box box gap single-dropdown input-text checkbox label title hyperlink-href p]]
+;; [re-com.dropdown :refer [filter-choices-by-keyword single-dropdown-args-desc]]
+ [youyesyet.ui-utils :as ui]
+))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;;;
+;;;; youyesyet.views.followup: followup-request view for youyesyet.
+;;;;
+;;;; This program is free software; you can redistribute it and/or
+;;;; modify it under the terms of the GNU General Public License
+;;;; as published by the Free Software Foundation; either version 2
+;;;; of the License, or (at your option) any later version.
+;;;;
+;;;; This program is distributed in the hope that it will be useful,
+;;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;;; GNU General Public License for more details.
+;;;;
+;;;; You should have received a copy of the GNU General Public License
+;;;; along with this program; if not, write to the Free Software
+;;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+;;;; USA.
+;;;;
+;;;; Copyright (C) 2016 Simon Brooke for Radical Independence Campaign
+;;;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+
+;;; The pattern from the re-com demo (https://github.com/Day8/re-com) is to have
+;;; one source file/namespace per view. Each namespace contains a function 'panel'
+;;; whose output is an enlive-style specification of the view to be redered.
+;;; I propose to follow this pattern. This file will (eventually) provide the followup-request view.
+
+;;; See https://github.com/simon-brooke/youyesyet/blob/master/doc/specification/userspec.md#followup-request-form
+
+(defn panel
+ "Generate the followup-request panel."
+ []
+ (js/console.log (str "Rendering follow-up form"))
+
+ (let [issue @(subscribe [:issue])
+ issues @(subscribe [:issues])
+ elector @(subscribe [:elector])
+ address @(subscribe [:address])
+ form (form/make-form {:elector (:id elector)
+ :issue (:id issue)})]
+ [:div
+ [:h1 "Followup Request"]
+ (let [selected-elector-id (r/atom (:id elector))
+ selected-issue (r/atom (:id issue))]
+ [:form {}
+ [:p.widget
+ [:label {:for "elector"} "Elector"]
+ [single-dropdown
+ :id elector
+ :choices (:electors address)
+ :model selected-elector-id
+ :label-fn #(:name %)]]
+ [:p.widget
+ [:label {:for "issue"} "Issue"]
+ [single-dropdown
+ :id issue
+ :choices (map #({:id % :label %}) (keys issues))
+ :model issue]]
+
+ ])]))
diff --git a/project.clj b/project.clj
index 8130d1e..7e9a72e 100644
--- a/project.clj
+++ b/project.clj
@@ -6,6 +6,7 @@
:dependencies [[org.clojure/clojure "1.8.0"]
[org.clojure/clojurescript "1.9.229" :scope "provided"]
[ring/ring-servlet "1.5.1"]
+ [lib-noir "0.9.9" :exclusions [org.clojure/tools.reader]]
[clj-oauth "1.5.5"]
[ch.qos.logback/logback-classic "1.2.2"]
[re-frame "0.9.2"]
@@ -58,19 +59,15 @@
[lein-bower "0.5.1"]
[lein-less "1.7.5"]]
- :bower-dependencies [
- ;; Problem with using boostrap and font-awsome from Bower: neither
- ;; of the distributed packages compile cleanly with less :-(
- ;; [bootstrap "2.3.1"]
- ;; [font-awesome "3.2.1"]
- [leaflet "0.7.3"]]
+ :bower-dependencies [[leaflet "0.7.3"]]
:cucumber-feature-paths ["test/clj/features"]
:hooks [leiningen.less]
:uberwar
- {:handler youyesyet.handler/app
+ {:prep-tasks ["compile" "bower" ["cljsbuild" "once" "min"]]
+ :handler youyesyet.handler/app
:init youyesyet.handler/init
:destroy youyesyet.handler/destroy
:name "youyesyet.war"}
@@ -88,20 +85,14 @@
:profiles
{:uberjar {:omit-source true
- :prep-tasks ["compile" ["cljsbuild" "once" "min"]]
+ :prep-tasks ["compile" ["bower" "install"] ["cljsbuild" "once" "min"]]
:cljsbuild
{:builds
{:min
{:source-paths ["src/cljc" "src/cljs" "env/prod/cljs"]
:compiler
- {:output-to "target/cljsbuild/public/js/app.js"
- :externs ["react/externs/react.js" "externs.js"]
- :optimizations :advanced
- :pretty-print false
- :closure-warnings
- {:externs-validation :off :non-standard-jsdoc :off}}}}}
-
-
+ {:optimizations :advanced
+ :pretty-print false}}}}
:aot :all
:uberjar-name "youyesyet.jar"
:source-paths ["env/prod/clj"]
@@ -118,7 +109,7 @@
[org.clojure/core.cache "0.6.5"]
[org.apache.httpcomponents/httpcore "4.4.6"]
[clj-webdriver/clj-webdriver "0.7.2"]
- [org.seleniumhq.selenium/selenium-server "3.3.1"]
+ [org.seleniumhq.selenium/selenium-server "3.3.1" :exclusions [org.seleniumhq.selenium/selenium-support]]
[doo "0.1.7"]
[binaryage/devtools "0.9.2"]
[figwheel-sidecar "0.5.9"]
@@ -135,6 +126,7 @@
:compiler
{:main "youyesyet.app"
:asset-path "/js/out"
+ :externs ["react/externs/react.js" "externs.js"]
:output-to "target/cljsbuild/public/js/app.js"
:output-dir "target/cljsbuild/public/js/out"
:source-map true
@@ -156,6 +148,7 @@
{:source-paths ["src/cljc" "src/cljs" "test/cljs"]
:compiler
{:output-to "target/test.js"
+ :externs ["react/externs/react.js" "externs.js"]
:main "youyesyet.doo-runner"
:optimizations :whitespace
:pretty-print true}}}}
diff --git a/resources/docs/docs.md b/resources/docs/docs.md
deleted file mode 100644
index 5930be1..0000000
--- a/resources/docs/docs.md
+++ /dev/null
@@ -1,35 +0,0 @@
-
-
-### Database Configuration is Required
-
-If you haven't already, then please follow the steps below to configure your database connection and run the necessary migrations.
-
-* Create the database for your application.
-* Update the connection URL in the `profiles.clj` file with your database name and login.
-* Run `lein run migrate` in the root of the project to create the tables.
-* Let `mount` know to start the database connection by `require`-ing youyesyet.db.core in some other namespace.
-* Restart the application.
-
-
-
-
-### Managing Your Middleware
-
-Request middleware functions are located under the `youyesyet.middleware` namespace.
-
-This namespace is reserved for any custom middleware for the application. Some default middleware is
-already defined here. The middleware is assembled in the `wrap-base` function.
-
-Middleware used for development is placed in the `youyesyet.dev-middleware` namespace found in
-the `env/dev/clj/` source path.
-
-### Here are some links to get started
-
-1. [HTML templating](http://www.luminusweb.net/docs/html_templating.md)
-2. [Accessing the database](http://www.luminusweb.net/docs/database.md)
-3. [Setting response types](http://www.luminusweb.net/docs/responses.md)
-4. [Defining routes](http://www.luminusweb.net/docs/routes.md)
-5. [Adding middleware](http://www.luminusweb.net/docs/middleware.md)
-6. [Sessions and cookies](http://www.luminusweb.net/docs/sessions_cookies.md)
-7. [Security](http://www.luminusweb.net/docs/security.md)
-8. [Deploying the application](http://www.luminusweb.net/docs/deployment.md)
diff --git a/resources/public/about.html b/resources/public/about.html
deleted file mode 100644
index 370071f..0000000
--- a/resources/public/about.html
+++ /dev/null
@@ -1,51 +0,0 @@
-
-
-
-
-
-
- About YouYesYet
-
-
-
-
- This isn't finished and doesn't work yet! This site is just a look-and-feel
- dummy.
-
-
- YouYesYet is a project to build a canvassing app for the new Scottish
- Independence Referendum. The source code is here. The specification
- is here.
-
-
- If we're going to get this working in time I cannot do it alone: I need help. Contact
- me by email or on on Twitter.
-
- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer tincidunt
- at ex id pretium. Proin nec ultricies mauris. Donec mattis, velit at commodo
- vehicula, nisi velit mattis justo, at tempor enim eros eget tortor. Quisque
- a porttitor lorem. Vestibulum tempus ex id sem laoreet, id fermentum enim
- pharetra. Cras diam ante, pulvinar sed pharetra sed, venenatis eget tellus.
- Quisque fermentum sem sed nulla mollis, et fermentum nisl pretium. Pellentesque
- porttitor interdum ultricies. Nunc ut accumsan leo, rutrum tempor tellus.
- Nam ultricies magna ipsum.
-
-
- Pellentesque in est rutrum, consectetur nisi vel, dictum felis. Quisque id
- elementum enim. Donec aliquet, massa id mattis semper, lectus elit scelerisque
- justo, quis dapibus tortor eros a erat. Vestibulum erat mauris, consectetur id
- condimentum ut, luctus vitae diam. Integer faucibus ultrices mi sed consequat.
- Aliquam lacinia sapien quis urna blandit, sed consectetur ligula gravida. Ut
- eleifend purus id mi vulputate faucibus ut quis risus. Donec dapibus finibus
- tincidunt. Nunc luctus libero tellus, eget porta diam lacinia vel. Pellentesque
- turpis nunc, venenatis vitae nisl eu, mollis pulvinar erat. Nulla scelerisque
- tellus eget ex hendrerit tincidunt.
-
-
- Duis tincidunt iaculis magna, ac rutrum velit congue quis. Maecenas feugiat
- efficitur sem, in hendrerit erat. Nunc congue, dui sit amet commodo faucibus,
- enim nisl feugiat nisl, a tincidunt massa metus nec nisi. Duis viverra nunc ut
- libero tempus, sed convallis elit dapibus. Sed venenatis condimentum odio, non
- elementum diam. Morbi fermentum metus justo, ac viverra dui fermentum at.
- Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere
- cubilia Curae; Aliquam erat volutpat.
-
-
- 
-
-