simon/walkmap
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Investigation of Dali performance issue

Browse source
This commit is contained in:
Simon Brooke 2020-05-25 12:11:06 +01:00
parent 33aa84a881
commit 387d817e9b
No known key found for this signature in database
GPG key ID: A7A4F18D1D4DF987
6 changed files with 250 additions and 51 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
doc/Dali-performance.ods Normal file
View file

Binary file not shown.

148
doc/dali-performance.md Normal file
View file

@ -0,0 +1,148 @@
# Dali performance
Notes written while trying to characterise the performance problem in Dali.
## Hypothesis one: it's the way I format the polygons that's the issue
Firstly, with both versions of `stl->svg` using the same version of `facet->svg-poly`, i.e. this one:
(defn- facet->svg-poly
[facet]
[:polygon
{:points (s/join " " (map #(str (:x %) "," (:y %)) (:vertices facet)))}])
we get this performance using the smaller `isle_of_man` map:
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :hiccup)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def hiccup (binary-stl-file->svg "resources/isle_of_man.stl" "resources/isle_of_man.svg")))
20-05-25 09:21:43 mason INFO [walkmap.svg:82] - Generating SVG for :hiccup renderer
20-05-25 09:21:43 mason INFO [walkmap.svg:96] - Emitting SVG with :hiccup renderer
"Elapsed time: 86.904891 msecs"
#'walkmap.svg/hiccup
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :dali)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def dali (binary-stl-file->svg "resources/isle_of_man.stl" "resources/isle_of_man.svg")))
20-05-25 09:22:17 mason INFO [walkmap.svg:82] - Generating SVG for :dali renderer
20-05-25 09:22:17 mason INFO [walkmap.svg:96] - Emitting SVG with :dali renderer
"Elapsed time: 890.863814 msecs"
#'walkmap.svg/dali
If we switch the Dali render to use my original version of `facet->svg-poly`, i.e. this one:
(defn- dali-facet->svg-poly
[facet]
(vec
(cons
:polygon
(map #(vec (list (:x %) (:y %))) (:vertices facet)))))
we get this performance:
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :hiccup)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def hiccup (binary-stl-file->svg "resources/isle_of_man.stl" "resources/isle_of_man.svg")))
20-05-25 09:35:33 mason INFO [walkmap.svg:82] - Generating SVG for :hiccup renderer
20-05-25 09:35:33 mason INFO [walkmap.svg:96] - Emitting SVG with :hiccup renderer
"Elapsed time: 84.09972 msecs"
#'walkmap.svg/hiccup
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :dali)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def dali (binary-stl-file->svg "resources/isle_of_man.stl" "resources/isle_of_man.svg")))
20-05-25 09:35:41 mason INFO [walkmap.svg:82] - Generating SVG for :dali renderer
20-05-25 09:35:41 mason INFO [walkmap.svg:96] - Emitting SVG with :dali renderer
"Elapsed time: 874.292007 msecs"
#'walkmap.svg/dali
No significant difference in performance.
If we generate but don't render, we get this:
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :hiccup)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def hiccup (binary-stl-file->svg "resources/isle_of_man.stl")))
20-05-25 09:37:44 mason INFO [walkmap.svg:82] - Generating SVG for :hiccup renderer
"Elapsed time: 52.614707 msecs"
#'walkmap.svg/hiccup
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :dali)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (def dali (binary-stl-file->svg "resources/isle_of_man.stl")))
20-05-25 09:38:07 mason INFO [walkmap.svg:82] - Generating SVG for :dali renderer
"Elapsed time: 49.891043 msecs"
#'walkmap.svg/dali
This implies that the problem is not in the way polygons are formatted.
The difference between the two versions of `facet->svg-poly` is as follows:
### New version, works with both Hiccup and Dali:
walkmap.svg=> (def stl (decode-binary-stl "resources/isle_of_man.stl"))
#'walkmap.svg/stl
walkmap.svg=> (def facet (first (:facets stl)))
#'walkmap.svg/facet
walkmap.svg=> (pprint facet)
{:normal {:x -0.0, :y 0.0, :z 1.0},
:vertices
[{:x 3.0, :y 1.0, :z 1.0}
{:x 2.0, :y 3.0, :z 1.0}
{:x 0.0, :y 0.0, :z 1.0}],
:abc 0}
nil
walkmap.svg=> (pprint (facet->svg-poly facet))
[:polygon {:points "3.0,1.0 2.0,3.0 0.0,0.0"}]
nil
In other words, the new version constructs the `:points` attribute of the `:polygon` tag by string concatenation, and the renderer just needs to output it.
### Older version, works with Dali only:
walkmap.svg=> (pprint (dali-facet->svg-poly facet))
[:polygon [3.0 1.0] [2.0 3.0] [0.0 0.0]]
nil
This means that the renderer is actually doing more work, since it has to compose the `:points` attribute itself; nevertheless there doesn't seem to be an increased time penalty.
### Conclusion
It doesn't seem that formatting the polygons is the issue.
## Hypothesis two: Dali renderer scales non-linearly with number of objects drawn
To test this, we need some otherwise-similar test files with different numbers of objects:
walkmap.svg=> (count (:facets stl))
4416
walkmap.svg=> (def small-stl (assoc stl :facets (take 400 (:facets stl))))
#'walkmap.svg/small-stl
walkmap.svg=> (count (:facets small-stl))
400
walkmap.svg=> (def large-stl (decode-binary-stl "../the-great-game/resources/maps/heightmap.stl"))
#'walkmap.svg/large-stl
walkmap.svg=> (count (:facets large-stl))
746585
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :dali)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (dali.io/render-svg (stl->svg small-stl) "dali-small.svg"))
20-05-25 10:12:25 mason INFO [walkmap.svg:92] - Generating SVG for :dali renderer
"Elapsed time: 32.55506 msecs"
nil
walkmap.svg=> (def ^:dynamic *preferred-svg-render* :hiccup)
#'walkmap.svg/*preferred-svg-render*
walkmap.svg=> (time (spit "hiccup-small.svg" (hiccup.core/html (stl->svg small-stl))))
20-05-25 10:14:07 mason INFO [walkmap.svg:92] - Generating SVG for :hiccup renderer
"Elapsed time: 10.026369 msecs"
So we have
| | Dali | | Hiccup | | |
| ----------- | ---------------- | ----------- | ------------ | ----------- | ------------------- |
| # of facets | time (msecs) | objets/msec | time (msecs) | objets/msec | ratio (Dali/Hiccup) |
| ----------- | ---------------- | ----------- | ------------ | ----------- | --------------------|
| 400 | 32.55506 | 12.29 | 10.026369 | 39.89 | 3.35 |
| 4416 | 874.292007 | 5.05 | 84.09972 | 52.51 | 10.40 |
| 746585 | 29,695,695.61 | 0.03 | 16724.848222 | 44.64 | 1775.54 |
### Conclusion
What we're seeing is that Hiccup renders more or less linearly by the number of objects (bear in mind that all of these objects are triangles, so essentially equally complex to render), whereas trhe performance of Dali degrades significantly as the number of objects increases.

3
project.clj
View file

@ -18,7 +18,8 @@
:license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
:url "https://www.eclipse.org/legal/epl-2.0/"}
:plugins [[lein-cloverage "1.1.1"]
[lein-codox "0.10.7"]]
[lein-codox "0.10.7"]
[lein-gorilla "0.4.0"]]
:release-tasks [["vcs" "assert-committed"]
["change" "version" "leiningen.release/bump-version" "release"]
["vcs" "commit"]

41
src/walkmap/core.clj
View file

@ -1,44 +1,9 @@
(ns walkmap.core
"At this stage, primarily utility functions dealing with stereolithography
(STL) files. Not a stable API yet!"
"This namespace mostly gets used as a scratchpad for ideas which haven't yet
solidified."
(:require [clojure.java.io :as io :refer [file output-stream input-stream]]
[clojure.string :as s]
[hiccup.core :refer [html]]
[me.raynes.fs :as fs]
[taoensso.timbre :as l :refer [info error spy]]
[walkmap.stl :refer [decode-binary-stl]]
[walkmap.svg :refer [stl->svg]]))
(def ^:dynamic *sea-level*
"The sea level on heightmaps we're currently handling. If characters are to
be able to swin in the sea, we must model the sea bottom, so we need
heightmaps which cover at least the continental shelf. However, the sea
bottom is not walkable territory and can be culled from walkmaps.
**Note** must be a floating point number. `(= 0 0.0)` returns `false`!"
0.0)
(defn ocean?
"Of a `facet`, is the altitude of every vertice equal to `*sea-level*`?"
[facet]
(every?
#(= % *sea-level*)
(map :z (:vertices facet))))
(defn cull-ocean-facets
"Ye cannae walk on water. Remove all facets from this `stl` structure which
are at sea level."
[stl]
(assoc stl :facets (remove ocean? (:facets stl))))
(defn binary-stl-file->svg
"Given only an `in-filename`, parse the indicated file, expected to be
binary STL, and return an equivalent SVG structure. Given both `in-filename`
and `out-filename`, as side-effect write the SVG to the indicated output file."
([in-filename]
(stl->svg (cull-ocean-facets (decode-binary-stl in-filename))))
([in-filename out-filename]
(let [s (binary-stl-file->svg in-filename)]
(spit out-filename (html s))
s)))
[taoensso.timbre :as l :refer [info error spy]]))

24
src/walkmap/ocean.clj Normal file
View file

@ -0,0 +1,24 @@
(ns walkmap.ocean
"Deal with (specifically, at this stage, cull) ocean areas")
(def ^:dynamic *sea-level*
"The sea level on heightmaps we're currently handling. If characters are to
be able to swin in the sea, we must model the sea bottom, so we need
heightmaps which cover at least the continental shelf. However, the sea
bottom is not walkable territory and can be culled from walkmaps.
**Note** must be a floating point number. `(= 0 0.0)` returns `false`!"
0.0)
(defn ocean?
"Of a `facet`, is the altitude of every vertice equal to `*sea-level*`?"
[facet]
(every?
#(= % *sea-level*)
(map :z (:vertices facet))))
(defn cull-ocean-facets
"Ye cannae walk on water. Remove all facets from this `stl` structure which
are at sea level."
[stl]
(assoc stl :facets (remove ocean? (:facets stl))))

85
src/walkmap/svg.clj
View file

@ -3,15 +3,68 @@
later, other geometry files of interest to us) as scalable vector graphics
(SVG)."
(:require [clojure.string :as s]
[dali.io :as neatly-folded-clock]
[hiccup.core :refer [html]]
[taoensso.timbre :as l :refer [info error spy]]
[walkmap.ocean :refer [cull-ocean-facets]]
[walkmap.polygon :refer [polygon?]]
[walkmap.stl :refer [decode-binary-stl]]
[walkmap.vertex :refer [vertex?]]))
(def ^:dynamic *preferred-svg-render*
"Mainly for debugging dali; switch SVG renderer to use. Expected values:
`:dali`, `:hiccup`."
:dali)
(defn- facet->svg-poly
;; When we use this version of facet->svg-poly with the Dali renderer, it's
;; still (for the isle_of_man map) 10 times slower than hiccup, also using
;; this version (890.863814 msecs vs 86.904891 msecs
[facet]
[:polygon
{:points (s/join " " (map #(str (:x %) "," (:y %)) (:vertices facet)))}])
(defn- dali-facet->svg-poly
[facet]
(vec
(cons
:polygon
(map #(vec (list (:x %) (:y %))) (:vertices facet)))))
(defn dali-stl->svg
"Format this `stl` as SVG for the `hiccup` renderer on a page with these
bounds."
[stl minx maxx miny maxy]
[:dali/page
{:xmlns "http://www.w3.org/2000/svg"
:version "1.2"
:width (- maxx minx)
:height (- maxy miny)
:viewBox (s/join " " (map str [minx miny maxx maxy]))}
(vec
(cons
:g
(map
dali-facet->svg-poly
(:facets stl))))])
(defn hiccup-stl->svg
"Format this `stl` as SVG for the `hiccup` renderer on a page with these
bounds."
[stl minx maxx miny maxy]
[:svg
{:xmlns "http://www.w3.org/2000/svg"
:version "1.2"
:width (- maxx minx)
:height (- maxy miny)
:viewBox (s/join " " (map str [minx miny maxx maxy]))}
(vec
(cons
:g
(map
facet->svg-poly
(:facets stl))))])
(defn stl->svg
"Convert this in-memory `stl` structure, as read by `decode-binary-stl`, into
an in-memory hiccup representation of SVG structure, and return it."
@ -36,15 +89,23 @@
(map
#(reduce max (map :y (:vertices %)))
(:facets stl)))]
[:svg
{:xmlns "http://www.w3.org/2000/svg"
:version "1.2"
:width (- maxx minx)
:height (- maxy miny)
:viewBox (s/join " " (map str [minx miny maxx maxy]))}
(vec
(cons
:g
(map
facet->svg-poly
(:facets stl))))]))
(l/info "Generating SVG for " *preferred-svg-render* " renderer")
(case *preferred-svg-render*
:hiccup (hiccup-stl->svg stl minx maxx miny maxy)
:dali (dali-stl->svg stl minx maxx miny maxy)
(throw (Exception. "Unexpected renderer value: " *preferred-svg-render*)))))
(defn binary-stl-file->svg
"Given only an `in-filename`, parse the indicated file, expected to be
binary STL, and return an equivalent SVG structure. Given both `in-filename`
and `out-filename`, as side-effect write the SVG to the indicated output file."
([in-filename]
(stl->svg (cull-ocean-facets (decode-binary-stl in-filename))))
([in-filename out-filename]
(let [s (binary-stl-file->svg in-filename)]
(l/info "Emitting SVG with " *preferred-svg-render* " renderer")
(case *preferred-svg-render*
:dali (neatly-folded-clock/render-svg s out-filename)
:hiccup (spit out-filename (html s))
(throw (Exception. "Unexpected renderer value: " *preferred-svg-render*)))
s)))