Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:[wip] minor tweaks, resume development
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | master | trunk
Files: files | file ages | folders
SHA3-256: 19689689ea94b8d2afff611fd8389b20a6f02e82a34e9b47a967c60882f5bbe4
User & Date: jack 2024-05-14 08:13:53
Context
2024-05-14
10:45
add a definition of done check-in: 7b66e5d647 user: jack tags: master, trunk
08:13
[wip] minor tweaks, resume development check-in: 19689689ea user: jack tags: master, trunk
2023-05-17
16:07
Clear separation of trainining passes (resolves fixme) check-in: 10fe2857c0 user: jack tags: master, trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to Code/dataset-and-aesthetics.lisp.

︙︙
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
                               (funcall ,cont ,@lambda-list))
                            `(lambda (,cont ,raw-point)
                               (let ((,point-values (point-values ,raw-point)))
                                 (funcall ,cont ,@lambda-list))))))
               (values (compile nil fun) fun)))))

;;; MAYBE currently we access dataset-head and then map points and use
;;; point-values. Perhaps a better approach will be passing header and elmenent
;;; accessor as functions.
(defun bind-data (aesthetics dataset &key (class <mapped-data>))
  (make-instance class :dataset dataset :aesthetics aesthetics))

;;; Binding data conses a new vector so it may be potentially space and time
;;; consuming operation. We hide that behind the protocol so an alternate
;;; appraoch is possible.







|







88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
                               (funcall ,cont ,@lambda-list))
                            `(lambda (,cont ,raw-point)
                               (let ((,point-values (point-values ,raw-point)))
                                 (funcall ,cont ,@lambda-list))))))
               (values (compile nil fun) fun)))))

;;; MAYBE currently we access dataset-head and then map points and use
;;; point-values. Perhaps a better approach will be passing header and element
;;; accessor as functions.
(defun bind-data (aesthetics dataset &key (class <mapped-data>))
  (make-instance class :dataset dataset :aesthetics aesthetics))

;;; Binding data conses a new vector so it may be potentially space and time
;;; consuming operation. We hide that behind the protocol so an alternate
;;; appraoch is possible.
︙︙

Changes to Code/plot.lisp.

︙︙
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
  (dolist (layer (layers plot))
    #+dont-retrain-layers 
    (unless (trained-dataset layer)
      ;; good for rendering, bad for mutations
      (train-layer plot layer))
    (train-layer plot layer)))

(defun train-facets (plot)
  (dolist (layer (layers plot))
    (compute-facets plot layer)
    (do-facets (facet plot)
      (let* ((dataset (dataset facet))
             (scales (scales facet))
             (aesthetics (layer-aesthetics plot layer))
             (stat-tr (statistical-transformation layer))







|







23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
  (dolist (layer (layers plot))
    #+dont-retrain-layers 
    (unless (trained-dataset layer)
      ;; good for rendering, bad for mutations
      (train-layer plot layer))
    (train-layer plot layer)))

(defun train-plot (plot)
  (dolist (layer (layers plot))
    (compute-facets plot layer)
    (do-facets (facet plot)
      (let* ((dataset (dataset facet))
             (scales (scales facet))
             (aesthetics (layer-aesthetics plot layer))
             (stat-tr (statistical-transformation layer))
︙︙
54
55
56
57
58
59
60


61
62

63
64
65
66
67
68
69
70
71
72




73
74
75
76
77
78
79
(c:define-presentation-method c:present ((plot <polyclot>)
                                         (type <polyclot>)
                                         stream
                                         (view <polyclot-view>)
                                         &rest args)
  (declare (ignore args))
  (with-theme (plot)


    (c:formatting-item-list (*standard-output* :x-spacing 20 :y-spacing 10 :n-columns 2)
      (do-facets (facet plot)

        (let ((coords (coords facet))
              (scales (scales facet)))
          (c:formatting-cell (*standard-output*)
            (c:draw-rectangle* *standard-output* 0 0 1 1
                               :ink c:+grey+
                               :filled nil)
            (with-coords (coords scales)
              (dolist (layer (layers plot))
                (render-geom layer coords (trained-dataset facet layer))))
            (render-guides plot coords)))))))





(defun polyclot (*plot* *standard-output*)
  (train-facets *plot*)
  (c:present *plot* <polyclot>
             :view (<polyclot-view>)
             :sensitive nil
             :allow-sensitive-inferiors t))







>
>
|

>
|
|
|
|
|
<
|
|
|
|
>
>
>
>


|




54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
(c:define-presentation-method c:present ((plot <polyclot>)
                                         (type <polyclot>)
                                         stream
                                         (view <polyclot-view>)
                                         &rest args)
  (declare (ignore args))
  (with-theme (plot)
    (let ((offset-x 0)
          (offset-y 0))
      ;;c:formatting-item-list (t :x-spacing 20 :y-spacing 10 :n-columns 2)
      (do-facets (facet plot)
        (c:with-translation (stream offset-x offset-y)
          (let ((coords (coords facet))
                (scales (scales facet)))
            (progn                       ;c:formatting-cell (*standard-output*)
              (c:draw-rectangle* *standard-output* 0 0 1 1
                                 :ink c:+grey+ :filled nil)

              (with-coords (coords scales)
                (dolist (layer (layers plot))
                  (render-geom layer coords (trained-dataset facet layer))))
              (render-guides plot coords))))
        (if (zerop offset-x)
            (incf offset-x)
            (setf offset-x 0
                  offset-y 1))))))

(defun polyclot (*plot* *standard-output*)
  (train-plot *plot*)
  (c:present *plot* <polyclot>
             :view (<polyclot-view>)
             :sensitive nil
             :allow-sensitive-inferiors t))

Changes to Code/theme.lisp.

︙︙
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
;;; FIXME can we leverege McCLIM space requirement and layout protocols?

(defgeneric call-with-theme (theme cont))

;;; Estabilishes a context for drawing at the current cursor position as a unit
;;; box [0 0 1 1] and adds the plot decorations.
(defmethod call-with-theme ((theme <theme>) cont)
  (c:surrounding-output-with-border (t :filled t :ink c:+white+ :outline-ink c:+deep-sky-blue+)
    (when (climi::null-bounding-rectangle-p theme)
      (multiple-value-bind (cx cy) (c:stream-cursor-position *standard-output*)
        (setf (c:rectangle-edges* theme)
              (values cx cy (+ cx (width theme)) (+ cy (height theme))))))
    ;; for debugging
    (c:draw-design *standard-output* theme :filled nil :ink c:+deep-pink+ :line-dashes t)
    (c:with-bounding-rectangle* (x0 y0 :width width :height height) theme
      
      (c:with-translation (t x0 y0)
        (c:with-scaling (t width height)
          (funcall cont))))))

(defmacro with-theme ((theme) &body body)
  `(call-with-theme ,theme (lambda () ,@body)))







<
|
|
|
|
<
<
|
<
|
|
|



38
39
40
41
42
43
44

45
46
47
48


49

50
51
52
53
54
55
;;; FIXME can we leverege McCLIM space requirement and layout protocols?

(defgeneric call-with-theme (theme cont))

;;; Estabilishes a context for drawing at the current cursor position as a unit
;;; box [0 0 1 1] and adds the plot decorations.
(defmethod call-with-theme ((theme <theme>) cont)

  (when (climi::null-bounding-rectangle-p theme)
    (multiple-value-bind (cx cy) (c:stream-cursor-position *standard-output*)
      (setf (c:rectangle-edges* theme)
            (values cx cy (+ cx (width theme)) (+ cy (height theme))))))


  (c:with-bounding-rectangle* (x0 y0 :width width :height height) theme

    (c:with-translation (t x0 y0)
      (c:with-scaling (t width height)
        (funcall cont)))))

(defmacro with-theme ((theme) &body body)
  `(call-with-theme ,theme (lambda () ,@body)))

Changes to README.org.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#+title: Polyclot
#+author: Daniel Kochmański

* Introduction

Polyclot is a CLIM library for plotting data inspired by ggplot2 and the
article "A Layered Grammar of Graphics" by Hadley Wickham. It is extended with
a possibility to interact with, and to incrementally modify plots.
Abstractions used should gravitate towards CLIM (not ggplot2) where feasilbe.

* Description

A single plot is composed of:
- a default dataset and its mapping to aesthetics
- one or more layers each having a geom
- one scale for each aesthetic
- a statistical transformation
- a coordinate system
- a facet specification

Additionally non-data parts are specified by the theme. The theme defines
things the size of the plot, margins, the title, colors, text styles etc.

Faceting allows grouping data into separate datasets that are displayed in
separate "boxes". There are three types of facets: none, wrap and grid.

Coordinate system maps the position of objects onto the plane of the plot.
Each facet will have a slightly different coordinate system at least to
account for the translation between them. The user local coordinates for
drawing are always a rectangle (0 0 1 1).

Scales are used to map a data point to its aesthetic attribues. Each used
aesthetic has exactly one scale associted. There are three groups of scales:

- position scales - guides are axes
- color scales - guides are legends
- other scales - guides are legends

* Developer Manual






|
|
|
<












|





|
<
|


|







1
2
3
4
5
6
7
8

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

28
29
30
31
32
33
34
35
36
37
38
#+title: Polyclot
#+author: Daniel Kochmański

* Introduction

Polyclot is a CLIM library for plotting data inspired by ggplot2 and the article
"A Layered Grammar of Graphics" by Hadley Wickham. Additionally we extend it
with a possibility to interact and incrementally modify plots.


* Description

A single plot is composed of:
- a default dataset and its mapping to aesthetics
- one or more layers each having a geom
- one scale for each aesthetic
- a statistical transformation
- a coordinate system
- a facet specification

Additionally non-data parts are specified by the theme. The theme defines
properties of the plot like its size, margins, the title, colors, etc.

Faceting allows grouping data into separate datasets that are displayed in
separate "boxes". There are three types of facets: none, wrap and grid.

Coordinate system maps the position of objects onto the plane of the plot.
Facets will have different coordinate system, but local coordinates for drawing

are always a rectangle (0 0 1 1).

Scales are used to map a data point to its aesthetic attribues. Each used
aesthetic has exactly one scale associted. There are three types of scales:

- position scales - guides are axes
- color scales - guides are legends
- other scales - guides are legends

* Developer Manual

︙︙
82
83
84
85
86
87
88













































89
90
91
92
93
94
95
96
97

98
99
100
101
102
103
104
|----------+-------------+------------------------------------------------------|
| labels   | :label      | text on the plot                                     |
| hjust    | :align-x    | text alignment (:left :center :right)                |
| vjust    | :align-y    | text alignment (:top :center :baseline :bottom)      |

* Development notes                                                :noexport:














































** Progress

As of [2022-12-11 nie] we have:

- coord: cartesian
- geoms: point, line and bar
- stats: identity and count
- adjut: jitter
- scales: {continuous, discrete, binned} x {x, y}


Dataset is for now parsed from a list (car contains headers) into an instance of
~<mapped-data>~, where data-array is an extensible vector of ~<row>~ instances.
Aesthetics are slots in each ~<row>~ - this isn't efficient nor extensible.

The API for mapping aesthetics is IMO good as it is now (and provides sane
defaults), i.e: ~:aesthetics (list :x "x" :y 3 :ink #'random-ink)~. I think







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


|






>







80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|----------+-------------+------------------------------------------------------|
| labels   | :label      | text on the plot                                     |
| hjust    | :align-x    | text alignment (:left :center :right)                |
| vjust    | :align-y    | text alignment (:top :center :baseline :bottom)      |

* Development notes                                                :noexport:

** Architecture

The central piece is the instance of the class <polyclot>. It has the following
slots:

- dataset :: raw dataset [fixme] document dataset-head, dataset-body, streams
- aesthetics :: default mapping between data and aesthetics (inherited [layers])
- layers :: each layer is a serie of data that is placed on the plot (geoms)
- scales :: scales are computed based on layers, and then they transform data to
  match the desired viewport and the data type
- coords :: the overall coordiante system for the plot (i.e cartesian)
- faceting :: grouping of data into multiple plots based on some characteristic
- theme :: theme adds all the fluff around actual plot and sets initial coords

A newly created <polyclot> object, before being plotted, needs to be trained.
This is because we don't know anything yet about the data -- for example its
range is unknown. Since data is faceted, we need to split the data first. The
first step that goes over all data is called training:

#+begin_src lisp
  (defun train-plot (plot)
    (map nil #'train-layer (layers plot)))

  (defun train-layer (plot layer)
    (compute-facets! plot layer)
    (map nil #'train-facet (facets plot)))

  (defun train-facet (plot facet)
    (analyze-data)
    (update-facet))
#+end_src

After the plot is trained, we can proceed to drawing it. To draw a <polyclot>
object on the stream it is enough to call ~(polyclot *plot* *clim-stream*)~.
Rendering of the plot works as follows:

1. The theme is used to show decorations, titles and legends
2. The theme estabilishes coordinates for the whole plot [0 0 1 1]
3. Each facet estabilishes coordinates for the individual facet and:
   a. renders all layers using the trained dataset for the facet
   b. renders guides

Rendering layers and guides depends on the ~coords~ and ~scales~. Both objects
depend on the facet.

** Progress

As of [2024-05-08 Wed] we have:

- coord: cartesian
- geoms: point, line and bar
- stats: identity and count
- adjut: jitter
- scales: {continuous, discrete, binned} x {x, y}
- facets: none, wrap, grid

Dataset is for now parsed from a list (car contains headers) into an instance of
~<mapped-data>~, where data-array is an extensible vector of ~<row>~ instances.
Aesthetics are slots in each ~<row>~ - this isn't efficient nor extensible.

The API for mapping aesthetics is IMO good as it is now (and provides sane
defaults), i.e: ~:aesthetics (list :x "x" :y 3 :ink #'random-ink)~. I think
︙︙
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
- [X] <cartesian> defaults to the screen orientation (should be graphics)
- [ ] <row> is an instance with fixed slots, we want to extend it (a vector?)
- [-] all geoms are very primitive often not even using relevant aes
  - [X] <geom-point>
  - [X] <geom-line>
  - [X] <geom-bar>
  - [ ] provide extra polishing(!)
- [6/9] implementation of scales and guides
  - [X] basic framework for scales is in place already
  - [X] scale-data :: transform the dataset (i.e log)
  - [X] train-scales :: combine ranges of all datasets in the plot
  - [X] continuous scale
  - [X] discrete scale
  - [X] binned scale
  - [-] implement for aesthetics other than x/y
    - [X] ink
    - [ ] line-style
  - [ ] when we work with discrete/binned positions we add .5 :: this does not
    seem to be correct, consider adjust-position dodge - it must fit numerous
    elements between [n; n+1], while automatic translation .5 fits only one
    element; should this translation be performed even by position-identity?
  - [ ] handle n/a values
- [1/4] position adjustment
  - [X] dodge bars







|






|

|







168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
- [X] <cartesian> defaults to the screen orientation (should be graphics)
- [ ] <row> is an instance with fixed slots, we want to extend it (a vector?)
- [-] all geoms are very primitive often not even using relevant aes
  - [X] <geom-point>
  - [X] <geom-line>
  - [X] <geom-bar>
  - [ ] provide extra polishing(!)
- [7/9] implementation of scales and guides
  - [X] basic framework for scales is in place already
  - [X] scale-data :: transform the dataset (i.e log)
  - [X] train-scales :: combine ranges of all datasets in the plot
  - [X] continuous scale
  - [X] discrete scale
  - [X] binned scale
  - [X] implement for aesthetics other than x/y
    - [X] ink
    - [X] line-style
  - [ ] when we work with discrete/binned positions we add .5 :: this does not
    seem to be correct, consider adjust-position dodge - it must fit numerous
    elements between [n; n+1], while automatic translation .5 fits only one
    element; should this translation be performed even by position-identity?
  - [ ] handle n/a values
- [1/4] position adjustment
  - [X] dodge bars
︙︙
190
191
192
193
194
195
196

197
198
199
200
201
202
203
News:
- [2022-11-08 Tue] moved the repository fossil.turtleware.eu/polyclot
- [2022-11-16 Wed] implemented binned scale (and fixed scale mapping)
- [2022-12-11 nie] improvements to scales, a simple listener tool (demo)
- [2022-12-13 wto] add colors, initial render of the color scale
- [2022-12-14 śro] position adjustment (dodge)
- [2023-05-17 Wed] the architecture has been changed to account for faceting


** Faceting

Facets are being implemented after a single plot abstraction has been defined
and that demands some architectural changes. Let's start with a description:

Faceting is an operation of splitting datasets into a few separate sets that are







>







234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
News:
- [2022-11-08 Tue] moved the repository fossil.turtleware.eu/polyclot
- [2022-11-16 Wed] implemented binned scale (and fixed scale mapping)
- [2022-12-11 nie] improvements to scales, a simple listener tool (demo)
- [2022-12-13 wto] add colors, initial render of the color scale
- [2022-12-14 śro] position adjustment (dodge)
- [2023-05-17 Wed] the architecture has been changed to account for faceting
- [2024-05-08 Wed] resuming development (almost a year! In train after els'24)

** Faceting

Facets are being implemented after a single plot abstraction has been defined
and that demands some architectural changes. Let's start with a description:

Faceting is an operation of splitting datasets into a few separate sets that are
︙︙
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
** Incremental plots

Ideally we'd like to be able to add new data to the existing plot. Consider:
- adding a new data point may change scales and influence whole facet
- it should be possible to add a set of new points without retraining repeatedly
- it would be cool to have some a GC possibility (i.e "remove older than")
- the order of data points matters (consider a line plot)


** Scales

When the plot is processed then each aesthetic has an associated a single
scale shared by all layers. Generally there are three major types of scales:

| scale type | input type | source range | range | result    |







<







256
257
258
259
260
261
262

263
264
265
266
267
268
269
** Incremental plots

Ideally we'd like to be able to add new data to the existing plot. Consider:
- adding a new data point may change scales and influence whole facet
- it should be possible to add a set of new points without retraining repeatedly
- it would be cool to have some a GC possibility (i.e "remove older than")
- the order of data points matters (consider a line plot)


** Scales

When the plot is processed then each aesthetic has an associated a single
scale shared by all layers. Generally there are three major types of scales:

| scale type | input type | source range | range | result    |
︙︙