simon/gaza-stats
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Reorganised code into three namespaces; implemented stats on donors.

Browse source
This commit is contained in:
Simon Brooke 2026-09-28 20:40:06 +01:00
parent 0e814583b7
commit 5dfa7fc3a3
4 changed files with 232 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

194
README.md
View file

@ -18,6 +18,10 @@ This is a very experimental, unfinished and unpolished library designed for use
At present several useful functions are implemented:
## gaza-stats.core
Functions to interrogate the [`gaza.onl`](https://gaza.onl/app.db) database.
### accounts-with-donations-by-date-range
```
@ -26,7 +30,7 @@ At present several useful functions are implemented:
Return a list of maps, each representing one account from the `gaza.onl` database, with a key `:donations`, whose value is a float representing the total donations received by that account between the dates `start-date` (exclusive) and `end-date` (inclusive). Date arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L134)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L108)
### active-accounts
@ -36,7 +40,7 @@ Return a list of maps, each representing one account from the `gaza.onl` databas
Return, as an integer, the number of accounts believed to have been active on this `date`, which should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L123)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L97)
### average-donations-received-by-month
@ -46,89 +50,19 @@ Return, as an integer, the number of accounts believed to have been active on th
Return data as a map keyed by the date of the first of each month between `start-date` and `end-date`; the values are maps with keys `:average`, `:count` and `:total`. Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L194)
### chart-accounts-with-donations-by-months-in-range
```
(chart-accounts-with-donations-by-months-in-range start-date end-date)
```
Return a chart with one line for each calendar month covering from `start-date` (inclusive) to `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`, showing donations as for `chart-donations-by-accounts-in-date-range`, q.v.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L168)
### chart-average-donations-received-by-month
```
(chart-average-donations-received-by-month start-date end-date)
```
Return a chart comparing the average donations received per account with the total number of active accounts per month between `start-date` and `end-date`. Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L216)
### chart-donations-as-percentage-in-date-range
```
(chart-donations-as-percentage-in-date-range start-date end-date)
```
Return a chart of donations received by individual accounts between `start-date` (exclusive) and `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`. Accounts are identified in the chart only by rank order.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L155)
### chart-donations-by-accounts-in-date-range
```
(chart-donations-by-accounts-in-date-range start-date end-date)
```
Return a chart of donations received by individual accounts between `start-date` (exclusive) and `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`. Accounts are identified in the chart only by rank order.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L145)
### chart-donations-by-weeks
```
(chart-donations-by-weeks start-date end-date key)
```
Create and return an XY chart of data on field represented by `key` of donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L83)
### chart-number-donations-by-weeks
```
(chart-number-donations-by-weeks start-date end-date)
```
Create and return an XY chart of data on number of donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L100)
### chart-total-donations-by-weeks
```
(chart-total-donations-by-weeks start-date end-date)
```
Create and return an XY chart of data on total donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L93)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L128)
### db
**TODO**: write docs
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L19)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L17)
### db-spec
Database specification in a format which can be used with either JDBC or Korma.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L12)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L10)
### enhance-account-with-donations
@ -138,7 +72,7 @@ Database specification in a format which can be used with either JDBC or Korma.
Take this `account_map`, which must at least have a valid value for the key `:campaign_url` and return a similar map including the key `:donations`, whose value is a float representing the total donations received by that campaign URL between the dates `sd` (exclusive) and `ed` (inclusive). Date arguments should be supplied as `java.time.LocalDate` objects.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L107)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L81)
### local-date->date
@ -150,7 +84,7 @@ Take this `account_map`, which must at least have a valid value for the key `:ca
Construct and return a new `java.util.Date` object from a `java.time.LocalDate` object. As usual, doing anything with Java time/date stuff is a complete bureaucratic nightmare.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L74)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L72)
### local-date?
@ -162,7 +96,7 @@ Construct and return a new `java.util.Date` object from a `java.time.LocalDate`
Is this `x` an instance of `java.time.LocalDate`?
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L23)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L21)
### normalise-date
@ -174,7 +108,7 @@ Is this `x` an instance of `java.time.LocalDate`?
More messing around with Java’s bizarre mess of date/time classes. We want our default date representation to be `java.time.LocalDate`, except, of course, when we don’t.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L28)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L26)
### total-donations-by-weeks
@ -182,7 +116,103 @@ More messing around with Java’s bizarre mess of date/time classes. We want our
Return a list of pairs `[java.time.LocalDate date, float amount]` representing the total of donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`. If `end-date` is not supplied, data for a single week will be returned.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L45)
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/core.clj#L43)
## gaza-stats.chart
Produce charts from data functions in other namespaces.
### chart-accounts-with-donations-by-months-in-range
```
(chart-accounts-with-donations-by-months-in-range start-date end-date)
```
Return a chart with one line for each calendar month covering from `start-date` (inclusive) to `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`, showing donations as for `chart-donations-by-accounts-in-date-range`, q.v.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L58)
### chart-average-donations-received-by-month
```
(chart-average-donations-received-by-month start-date end-date)
```
Return a chart comparing the average donations received per account with the total number of active accounts per month between `start-date` and `end-date`. Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L76)
### chart-donations-as-percentage-in-date-range
```
(chart-donations-as-percentage-in-date-range start-date end-date)
```
Return a chart of donations received by individual accounts between `start-date` (exclusive) and `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`. Accounts are identified in the chart only by rank order.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L45)
### chart-donations-by-accounts-in-date-range
```
(chart-donations-by-accounts-in-date-range start-date end-date)
```
Return a chart of donations received by individual accounts between `start-date` (exclusive) and `end-date` (inclusive). Arguments should be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`. Accounts are identified in the chart only by rank order.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L35)
### chart-donations-by-donor
`(chart-donations-by-donor start-date end-date key)``(chart-donations-by-donor start-date end-date key exclude-anons)`
Return a rank-ordered graph of the values of this `key` in the `donations-by-donor` data between this `start-date` and this `end-date`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L88)
### chart-donations-by-weeks
```
(chart-donations-by-weeks start-date end-date key)
```
Create and return an XY chart of data on field represented by `key` of donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L11)
### chart-number-donations-by-weeks
```
(chart-number-donations-by-weeks start-date end-date)
```
Create and return an XY chart of data on number of donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L28)
### chart-total-donations-by-weeks
```
(chart-total-donations-by-weeks start-date end-date)
```
Create and return an XY chart of data on total donations made between `start-date` and `end-date`. Arguments may be supplied as `java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`.
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/chart.clj#L21)
## gaza-stats.donors
Statistics on donors.
### donations-by-donor
```
(donations-by-donor start-date end-date)
```
Return a sequence of maps, one representing each donor who has donated between `start-date` and `end-date`, each with keys `:count`, `:donor` and `:total`
[view source](https://git.journeyman.cc/simon/gaza-stats/src/branch/main/src/gaza_stats/donors.clj#L6)
### Viewing charts
@ -207,6 +237,10 @@ The graphics format will be taken from the file extension, and at least all of `
This code is designed to explore data in a [SQLite](https://www.sqlite.org/) database which it expects to be called `app.db` and find in its `resources` directory. This database is not included in this repository but may be downloaded from [here](https://gaza.onl/app.db).
## NOTE: Currency of `amount` column
The `campaign_donations` table has a column `amount`. I don't have any documentation on what currency the column is denominated in, but by inspecting my own donations, which I know to be in Pounds Sterling, I infer the column is probably denominated in US Dollare.
## NOTE: Use of SQL Korma
I used SQL Korma in this project because I know it and I like it, but it is no longer maintained and its website is no longer online. The last version I could find on the Internet Archive is [here](https://web.archive.org/web/20190223025406/http://www.sqlkorma.com/). The source code is on [Github](https://github.com/korma/Korma). It might be better to rewrite so as not to use it, especially given that (at present) the SQL usage is trivial.

98
src/gaza_stats/chart.clj Normal file
View file

@ -0,0 +1,98 @@
(ns gaza-stats.chart
"Produce charts from data functions in other namespaces."
(:require [cljc.java-time.local-date :refer [is-before]]
[clojure.string :refer [capitalize starts-with?]]
[com.hypirion.clj-xchart :refer [xy-chart]]
[gaza-stats.core :refer [accounts-with-donations-by-date-range
average-donations-received-by-month local-date->date
normalise-date total-donations-by-weeks]]
[gaza-stats.donors :refer [donations-by-donor]]))
(defn chart-donations-by-weeks
"Create and return an XY chart of data on field represented by `key` of
donations made between `start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date key]
(let [data (total-donations-by-weeks start-date end-date)]
(xy-chart {(format "%s Donations" (capitalize (name key)))
[(map #(local-date->date (first %)) data)
(map #(:n (nth % 1)) data)]})))
(defn chart-total-donations-by-weeks
"Create and return an XY chart of data on total donations made between
`start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(chart-donations-by-weeks start-date end-date :total))
(defn chart-number-donations-by-weeks
"Create and return an XY chart of data on number of donations made between
`start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(chart-donations-by-weeks start-date end-date :n))
(defn chart-donations-by-accounts-in-date-range
"Return a chart of donations received by individual accounts between
`start-date` (exclusive) and `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`. Accounts are identified in the chart only by rank order."
[start-date end-date]
(let [data (accounts-with-donations-by-date-range start-date end-date)]
(xy-chart {(format "Donations between\n%s and\n%s" start-date end-date)
[(range (count data)) (sort (map :donations data))]})))
(defn chart-donations-as-percentage-in-date-range
"Return a chart of donations received by individual accounts between
`start-date` (exclusive) and `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`. Accounts are identified in the chart only by rank order."
[start-date end-date]
(let [data (accounts-with-donations-by-date-range start-date end-date)
donations (sort (map :donations data))
total (reduce + donations)
percentages (map #(* 100.0 (/ % total)) donations)]
(xy-chart {(format "Donations as percent\nof total donations\nbetween %s and\n%s" start-date end-date)
[(range (count data)) percentages]})))
(defn chart-accounts-with-donations-by-months-in-range
"Return a chart with one line for each calendar month covering from
`start-date` (inclusive) to `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`, showing donations as for
`chart-donations-by-accounts-in-date-range`, q.v."
[start-date end-date]
(let [sd (normalise-date start-date "2024-01-01")
ed (normalise-date end-date nil)]
(loop [s (.withDayOfMonth sd 1)
i (.withDayOfMonth (.plusMonths sd 1) 1)
m (sorted-map)]
(let [data (accounts-with-donations-by-date-range s i)
m' (assoc m (format "Donations between\n%s and\n%s" s i)
[(range (count data)) (sort (map :donations data))])]
(if (is-before ed i) (xy-chart m')
(recur i (.withDayOfMonth (.plusMonths i 1) 1) m'))))))
(defn chart-average-donations-received-by-month
"Return a chart comparing the average donations received per account with
the total number of active accounts per month between `start-date` and
`end-date`. Arguments should be supplied as `java.time.LocalDate` objects,
or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(let [data (average-donations-received-by-month start-date end-date)]
(xy-chart {(format "Average donations\nper active account\nbetween\n%s and\n%s" start-date end-date)
[(map #(local-date->date %) (keys data)) (map :average (vals data))]
(format "Active accounts\nbetween\n%s and\n%s" start-date end-date)
[(map #(local-date->date %) (keys data)) (map :count (vals data))]})))
(defn chart-donations-by-donor
"Return a rank-ordered graph of the values of this `key` in the
`donations-by-donor` data between this `start-date` and this `end-date`."
([start-date end-date key]
(chart-donations-by-donor start-date end-date key false))
([start-date end-date key exclude-anons]
(let [data (donations-by-donor start-date end-date)
data' (if exclude-anons (remove #(let [d (:donor %)](or (nil? d) (starts-with? d "Anon"))) data) data)]
(xy-chart {(format "%s dontations by donor%s" (capitalize (name key)) (when exclude-anons "\n(excluding anons)"))
[(range (count data'))
(sort (map key data'))]}))))

80
src/gaza_stats/core.clj
View file

@ -1,8 +1,6 @@
(ns gaza-stats.core
"Functions to interrogate the [`gaza.onl`](https://gaza.onl/app.db) database."
(:require [clojure.string :refer [capitalize]]
[cljc.java-time.local-date :as ld]
[com.hypirion.clj-xchart :refer [xy-chart]]
(:require [cljc.java-time.local-date :as ld]
[korma.core :refer [as-sql fields join select where]]
[korma.db :refer [defdb sqlite3]])
(:import [java.util Date]
@ -80,30 +78,6 @@
(.toInstant (.atStartOfDay ~x)
(ZoneOffset/ofHours 0))))
(defn chart-donations-by-weeks
"Create and return an XY chart of data on field represented by `key` of
donations made between `start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date key]
(let [data (total-donations-by-weeks start-date end-date)]
(xy-chart {(format "%s Donations" (capitalize (name key)))
[(map #(local-date->date (first %)) data)
(map #(:n (nth % 1)) data)]})))
(defn chart-total-donations-by-weeks
"Create and return an XY chart of data on total donations made between
`start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(chart-donations-by-weeks start-date end-date :total))
(defn chart-number-donations-by-weeks
"Create and return an XY chart of data on number of donations made between
`start-date` and `end-date`. Arguments may be supplied as
`java.time.LocalDate` objects, or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(chart-donations-by-weeks start-date end-date :n))
(defn enhance-account-with-donations
"Take this `account_map`, which must at least have a valid value for the
key `:campaign_url` and return a similar map including the key `:donations`,
@ -142,46 +116,6 @@
ed (normalise-date end-date (ld/now))]
(map #(enhance-account-with-donations % sd ed) (active-accounts sd))))
(defn chart-donations-by-accounts-in-date-range
"Return a chart of donations received by individual accounts between
`start-date` (exclusive) and `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`. Accounts are identified in the chart only by rank order."
[start-date end-date]
(let [data (accounts-with-donations-by-date-range start-date end-date)]
(xy-chart {(format "Donations between\n%s and\n%s" start-date end-date)
[(range (count data)) (sort (map :donations data))]})))
(defn chart-donations-as-percentage-in-date-range
"Return a chart of donations received by individual accounts between
`start-date` (exclusive) and `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`. Accounts are identified in the chart only by rank order."
[start-date end-date]
(let [data (accounts-with-donations-by-date-range start-date end-date)
donations (sort (map :donations data))
total (reduce + donations)
percentages (map #(* 100.0 (/ % total)) donations)]
(xy-chart {(format "Donations as percent\nof total donations\nbetween %s and\n%s" start-date end-date)
[(range (count data)) percentages]})))
(defn chart-accounts-with-donations-by-months-in-range
"Return a chart with one line for each calendar month covering from
`start-date` (inclusive) to `end-date` (inclusive). Arguments should be
supplied as `java.time.LocalDate` objects, or as strings in the format
`yyyy-mm-dd`, showing donations as for
`chart-donations-by-accounts-in-date-range`, q.v."
[start-date end-date]
(let [sd (normalise-date start-date "2024-01-01")
ed (normalise-date end-date (ld/now))]
(loop [s (.withDayOfMonth sd 1)
i (.withDayOfMonth (.plusMonths sd 1) 1)
m (sorted-map)]
(let [data (accounts-with-donations-by-date-range s i)
m' (assoc m (format "Donations between\n%s and\n%s" s i)
[(range (count data)) (sort (map :donations data))])]
(if (ld/is-before ed i) (xy-chart m')
(recur i (.withDayOfMonth (.plusMonths i 1) 1) m'))))))
;; this does not work and I don't know why
;; (as-sql '(select 'campaign_donations
@ -212,15 +146,3 @@
(if (ld/is-before ed i)
m'
(recur i (.withDayOfMonth (.plusMonths i 1) 1) m'))))))
(defn chart-average-donations-received-by-month
"Return a chart comparing the average donations received per account with
the total number of active accounts per month between `start-date` and
`end-date`. Arguments should be supplied as `java.time.LocalDate` objects,
or as strings in the format `yyyy-mm-dd`."
[start-date end-date]
(let [data (average-donations-received-by-month start-date end-date)]
(xy-chart {(format "Average donations\nper active account\nbetween\n%s and\n%s" start-date end-date)
[(map #(local-date->date %) (keys data)) (map :average (vals data))]
(format "Active accounts\nbetween\n%s and\n%s" start-date end-date)
[(map #(local-date->date %) (keys data)) (map :count (vals data))]})))

19
src/gaza_stats/donors.clj Normal file
View file

@ -0,0 +1,19 @@
(ns gaza-stats.donors
"Statistics on donors."
(:require [gaza-stats.core :refer [db normalise-date]]
[korma.core :refer [aggregate as-sql fields join order select where]]))
(defn donations-by-donor
"Return a sequence of maps, one representing each donor who has donated
between `start-date` and `end-date`, each with keys `:count`, `:donor` and
`:total`"
[start-date end-date]
(let [sd (normalise-date start-date "2024-01-01")
ed (normalise-date end-date nil)]
(select 'campaign_donations
(fields :donor)
(aggregate (sum :amount) :total :donor)
(aggregate (count :id) :count :donor)
(where (and {:created_at [> sd]}
{:created_at [<= ed]}))
(order :total))))