diff --git a/doc/specification/competitors.md b/doc/specification/competitors.md new file mode 100644 index 0000000..8fa7dd0 --- /dev/null +++ b/doc/specification/competitors.md @@ -0,0 +1,23 @@ +# Competitor Analysis + +Obviously **You Yes Yet?** is my baby; I've put a lot of thought into it. At the time I started working on it I wasn't aware of any open source competitors; I did to a web search, and I emailed the Bernie Sanders campaign to see whether their widely admired tools were open source. I didn't find anything. + +However, I've just been pointed to [Vote Leave](http://www.voteleavetakecontrol.org/)'s [Vics](https://dominiccummings.wordpress.com/2016/10/29/on-the-referendum-20-the-campaign-physics-and-data-science-vote-leaves-voter-intention-collection-system-vics-now-available-for-all/) tool, and there may well be others. + +There is no room here for ego. What matters is that the Yes campaign gets the best available tool for the job. So it's important to do competitor analysis, and not to invest too much work into **You Yes Yet?** unless there's a realistic possibility of producing a tool which is better than any of the available alternatives. But it's also the case that by studying competitors we may find ways to improve the design of **You Yes Yet?**. + +## Vics + +Vics, the **Voter Intention Collection System**, is reputed to have been a significant factor in the successful campaign by Vote Leave to take Britain out of the EU. It has been [released](https://github.com/celestial-winter/vics) as open source under MIT licence, so it is unambiguously available for us to use. + +The architecture comprises a single-page app built using Angular talking to a server built in Java using the Spring framework. The database engine used is Postgres. [Jedis](https://github.com/xetorthio/jedis), a Java port of [Redis](https://redis.io/), is used as an in-memory data cache, server side. + +### Download and initial build + +I checked out the source from the GitHub repository, and following the instructions in the README created the database and ran a maven install process. Unfortunately, run as a normal user, when this process goes into its test sequence many tests fail unable to contact Jedis. I find it slightly worrying to run such a large and complex build as *root*, but as *root* it gets substantially further. The build still doesn't complete but it seems that it is closer to completion. + +The ironic point is that it fails because it depends on the JavaScript package manager [bower](https://bower.io/), and bower (very sensibly) refuses to run as *root*. I therefore made a small modification to the build script to allow it to run *bower* as root, but unfortunately that didn't solve the build problem; the jedis service was still not found where it was expected. + +This is difficult to diagnose; the exception is so deeply nested in framework code that no code from the actual Vics application appears on the stack dump, which makes it very hard to know where to start in debugging. + +So for tonight I've failed. I shall try again. diff --git a/doc/specification/scaling.md b/doc/specification/scaling.md index 5a8de4d..2444ddf 100644 --- a/doc/specification/scaling.md +++ b/doc/specification/scaling.md @@ -42,6 +42,22 @@ This means that the maximum number of transactions per second across Scotland is ## Spreading the load +### Caching and memoizing + +People typically go out canvassing in teams; each member of the team will need the same elector data. + +Glasgow has a population density of about 3,260 per Km^2; that means each half kilometer square has a maximum population of not much more than 1,000. Downloading 1,000 elector records at startup time is not infeasible. If we normalise data requests to a 100 metre square grid and serve records in 500 metre square chunks, all the members of the same team will request the same chunk of data. Also, elector data is not volatile. Therefore it makes sense to memoize requests for elector data. The app should only request fresh elector data when the device moves within 100 metres of the edge of the current 500 metre cell. + +Intention data is volatile: we'll want to update canvassers with fresh intention data frequently, because the other members of their team will be recording intention data as they work, and it's by seeing that intention data that the canvassers know which doors are still unchapped. So we don't want to cache intention data for very long. But nevertheless it still makes sense to deliver it in normalised 500 metre square chunks, because that means we can temporarily cache it server side and do not actually have to hit the database with many requests for the same data. + +Finally, issue data is not volatile over the course of a canvassing session, although it may change over a period of days. So issue data - all the current issues - should be fetched once at app startup time, and not periodically refreshed during a canvassing session. Also, of course, every canvasser will require exactly the same issue data (unless we start thinking of local or regional issues...?), so it absolutely makes sense to memoise requests for issue data. + +All this normalisation and memoisation reduces the number of read requests on the database. + +Note that [clojure.core.memoize](https://github.com/clojure/core.memoize) provides us with functions to create both size-limited, least-recently-used caches and duration limited, time-to-live caches. + +At 56 degrees north there are 111,341 metres per degree of latitude, 62,392 metres per degree of longitude. So a 100 metre box is about 0.0016 degrees east-west and .0009 degrees north-south. If we simplify that slightly (and we don't need square boxes, we need units of area covering a group of people working together) then we can take .001 of a degree in either direction which is computationally cheap. + ### Geographic sharding Volunteers canvassing simultaneously in the same street or the same locality need to see in near real time which dwellings have been canvassed by other volunteers, otherwise we'll get the same households canvassed repeatedly, which wastes volunteer time and annoys voters. So they all need to be sending updates to, and receiving updates from, the same server. But volunteers canvassing in Aberdeen don't need to see in near real time what is happening in Edinburgh. diff --git a/project.clj b/project.clj index 8130d1e..4fd0891 100644 --- a/project.clj +++ b/project.clj @@ -6,6 +6,7 @@ :dependencies [[org.clojure/clojure "1.8.0"] [org.clojure/clojurescript "1.9.229" :scope "provided"] [ring/ring-servlet "1.5.1"] + [lib-noir "0.9.9" :exclusions [org.clojure/tools.reader]] [clj-oauth "1.5.5"] [ch.qos.logback/logback-classic "1.2.2"] [re-frame "0.9.2"] diff --git a/resources/templates/login.html b/resources/templates/login.html new file mode 100644 index 0000000..e8685b4 --- /dev/null +++ b/resources/templates/login.html @@ -0,0 +1,26 @@ +{% extends "base-unauthenticated.html" %} +{% block big-links %} + +{% endblock %} +{% block content %} +

+ We're not going to do login in the long term; we're going to use oauth. + This is a temporary login form. +

+
+

+ + +

+

+ + +

+

+ + +

+
+{% endblock %} diff --git a/src/clj/youyesyet/routes/home.clj b/src/clj/youyesyet/routes/home.clj index 35d66da..b420e84 100644 --- a/src/clj/youyesyet/routes/home.clj +++ b/src/clj/youyesyet/routes/home.clj @@ -1,5 +1,6 @@ (ns youyesyet.routes.home - (:require [youyesyet.layout :as layout] + (:require [noir.util.route :as route] + [youyesyet.layout :as layout] [youyesyet.db.core :as db-core] [compojure.core :refer [defroutes GET POST]] [ring.util.http-response :as response] @@ -11,21 +12,38 @@ (defn call-me-page [request] (if request - ;; do something to store it in the database - (layout/render "call-me-accepted.html" (:params request)) + (do + ;; do something to store it in the database + (layout/render "call-me-accepted.html" (:params request))) (layout/render "call-me.html" {:title "Please call me!" - ;; TODO: Concerns need to be fetched from the database + ;; TODO: Issues need to be fetched from the database :concerns nil}))) (defn home-page [] (layout/render "home.html" {:title "You Yes Yet?"})) +(defn login-page + "This is very temporary. We're going to do authentication by oauth." + [request] + (let [params (keywordize-keys (:form-params request)) + username (:username params) + password (:password params) + redirect-to (or (:redirect-to params) "app")] + (if + (and (= username "test" (= password "test")) + (do + (session/put! :user username) + (response/redirect redirect-to)) + (layout/render "login.html" {:title "Please log in" :redirect-to redirect-to}))))) + (defroutes home-routes (GET "/" [] (home-page)) - (GET "/app" [] (app-page)) + (GET "/app" [] (route/restricted (app-page))) (GET "/call-me" [] (call-me-page nil)) (POST "/call-me" request (call-me-page request)) + (GET "/auth" request (login-page request)) + (POST "/auth" request (login-page request)) (GET "/notyet" [] (layout/render "notyet.html" {:title "Can we persuade you?"})) (GET "/supporter" [] (layout/render "supporter.html" diff --git a/src/cljs/youyesyet/core.cljs b/src/cljs/youyesyet/core.cljs index bbc546b..1bd80e0 100644 --- a/src/cljs/youyesyet/core.cljs +++ b/src/cljs/youyesyet/core.cljs @@ -23,12 +23,18 @@ (defn home-page [] (home/panel)) +(defn issues-page [] + (issues/panel)) (defn map-page [] (maps/panel)) +(defn request-page [] + (request/panel)) + (def pages {:home #'home-page + :issues #'issues-page :map #'map-page :about #'about-page}) diff --git a/src/cljs/youyesyet/views/issues.cljs b/src/cljs/youyesyet/views/issues.cljs index fede400..2c0e196 100644 --- a/src/cljs/youyesyet/views/issues.cljs +++ b/src/cljs/youyesyet/views/issues.cljs @@ -1,5 +1,6 @@ (ns youyesyet.views.issues - (:require [re-frame.core :refer [reg-sub]])) + (:require [re-frame.core :refer [reg-sub]] + [youyesyet.ui-utils :as ui])) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;; @@ -32,7 +33,32 @@ ;;; See https://github.com/simon-brooke/youyesyet/blob/master/doc/specification/userspec.md#issues-view +;;; The same panel, using re-frame, will display both the list of issues and the prompt +;;; text on a single issue. All the issues will be fetched once as a map at client load +;;; time + + +(def *issues* + ;;; this is a dummy for the map fetched at load-time + {"Currency" "Lorem ipsum dolar sit amet" + "Head of state" "Lorem ipsum dolar sit amet" + "NATO and defence" "Lorem ipsum dolar sit amet"}) + +(defn get-issues-fn + "This is a temporary dummy for the function which will pull the issues from + the server." + [] + *issues*) + +;;; By memoising the function we arange that it is called only once +(def get-issues (memoize get-issues-fn)) + (defn panel "Generate the issues panel." [] - []) + [:div.container {:id "main-container"} + (ui/back-link) + [:div {:id "issue-list"} + ] + (map (fn [k] (ui/big-link k k)) (keys (get-issues))) +])