Visualizing probability distributions across bivariate cyclic temporal granularities

class: center, middle, inverse, title-slide

# Visualizing probability distributions across <br> bivariate cyclic temporal granularities
### <br> Sayani Gupta <br><small><small> <i class='fab fa-github'></i> Sayani07     <i class='fab fa-twitter' style='color:#6CADDE'></i> SayaniGupta07 <br><font> <small><a href="https://sayanigupta-ows2020.netlify.com/" class="uri">https://sayanigupta-ows2020.netlify.com/</a></small>
### <br> <small> Time series analysis and econometrics <br> <small><small> Bernoulli-IMS One World Symposium 2020 <br> August 24, 2020 - August 28, 2020

---

background-image: url(images/title_page3.png)
background-size: cover
class: hide-slide-number split-70 title-slide
count: false

.column.shade_black[.content[

<br>

<br>

.bottom_abs.width100[

Joint work with Professor Dianne Cook and Rob Hyndman

Department of Econometrics and Business Statistics
<br>
Monash University
<br>
<span><i class="fas  fa-envelope faa-float animated "></i></span>  Sayani.Gupta@monash.edu

<br>
]

]]

---
class: center

# Motivation: Smart meter data for households

![](images/smart_allcust.gif)

---
class: top left

# Linear to cyclic

.pull-left[
.checked[
.smaller[
- **Cyclic time granularities:** exploring different periodicities e.g. hour-of-day, day-of-month or  hour-of-week, day-of-semester

- **Multiple observations for each level of cyclic granularity**

- summarize distribution of measured variables
]
]
]

.pull-right[
![](index_files/figure-html/linear2cyclic-.gif)
]
---
class: top left

# Summarize distribution

.pull-left[
.checked[
.smaller[
- **Cyclic time granularities:** exploring different periodicities e.g. hour-of-day, day-of-month or  hour-of-week, day-of-semester

- Multiple observations for each level of cyclic granularity

- **summarize distribution of measured variables**
]
]
]

.pull-right[
![](index_files/figure-html/allplot-.gif)
]
---
class: top left

# Data structure and graphical mapping

.left-column[
<br>
<br>
-  extension of tsibble data structure
-  choose any two cyclic granularities: 
`\(C_i = \{A_1, A_2, \dots, A_K\}\)` and `\(C_j = \{B_1, B_2, \dots, B_L\}\)`
- graphical mapping `\((C_i, C_j, v)\)`

- `\(^{N_C}P_2\)` displays
]

.right-column[

]
---
class:left, top

# Relationship of cyclic granularities

.pull-left[
**<span style="color:firebrick"> <i> Clashes</i>:** pairs leading to empty sets

]

.pull-right[
**<span style="color:firebrick"> <i> Harmonies</i>:** pairs leading to no empty sets

<img src="index_files/figure-html/noclash-1.png" width="100%" />
]

Still too many harmonies for display for large `\(N_C\)`
---
# Selecting harmonies with significant variation

.pull-left[
<img src="index_files/figure-html/question1-1.png" width="200%" />
]

.pull-right[
- significant differences between categories: <br> (a) no (b) yes
- Gestalt theory
- efficiently capture within and between group variation 
- compute threshold
 ]
---
# Rank selected harmonies <br> Median Maximum Pairwise Distance (MMPD)

.pull-left[
![](index_files/figure-html/questioncric-1.png)
]
.pull-right[
- rank harmonies (b) > (a)
- distances between probability distribution through Jensen-Shannon divergences
- normalizing constants through Fisher–Tippett–Gnedenko theorem
]
---
R package `gravitas`

.pull-left[
.animated.bounce[
<img src="images/gravitas_sticker.png" height=280px>
]
]

.pull-right[
* construction of cyclic granularities and cyclic calendar algebra 
* check feasibility through harmony/clash
* select significant harmonies and rank them
* recommend prospective probability distributions 
]
---
class: top left

## An example : Electricity smart meter data

<i><small>Data source</i></small> : [<small><i>Department of the Environment and Energy, Australia</i></small>](https://data.gov.au/dataset/4e21dea3-9b87-4610-94c7-15a8a77907ef)

```
## # A tsibble: 293,874 x 3 [30m] <UTC>
## # Key:       customer_id [10]
##    customer_id reading_datetime    general_supply_kwh
##    <chr>       <dttm>                           <dbl>
##  1 10006414    2012-02-10 08:00:00              0.141
##  2 10006414    2012-02-10 08:30:00              0.088
##  3 10006414    2012-02-10 09:00:00              0.078
##  4 10006414    2012-02-10 09:30:00              0.151
##  5 10006414    2012-02-10 10:00:00              0.146
##  6 10006414    2012-02-10 10:30:00              0.077
##  7 10006414    2012-02-10 11:00:00              0.052
##  8 10006414    2012-02-10 11:30:00              0.055
##  9 10006414    2012-02-10 12:00:00              0.055
## 10 10006414    2012-02-10 12:30:00              0.252
## # … with 293,864 more rows
```
---

# Cyclic granularities considered for analysis

```
## [1] "hour_day"   "hour_week"  "hour_month" "day_week"   "day_month" 
## [6] "week_month" "wknd_wday"
```
<br>
<br>

# Number of displays to analyze for exhaustive exploration

There are `\(^{7}P_2 = 42\)` pair of granularities to look at placing one on the x-axis and the other on facet.
---

# Select only harmonies (16 out of 42 selected)

<table class="table table-striped" style="width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;background-color: white !important;"> facet </th>
   <th style="text-align:left;background-color: white !important;"> x-axis </th>
   <th style="text-align:right;background-color: white !important;"> facet levels </th>
   <th style="text-align:right;background-color: white !important;"> x levels </th>
   <th style="text-align:right;background-color: white !important;"> MMPD </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.064 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.060 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.044 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.024 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.023 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.016 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.014 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.011 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 31 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.009 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.009 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.007 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.004 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> wknd_wday </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 2 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.003 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.003 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> week_month </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 5 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.001 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> hour_day </td>
   <td style="text-align:left;color: black !important;background-color: white !important;font-size: 18px;"> day_week </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 24 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 7 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;font-size: 18px;"> 0.001 </td>
  </tr>
</tbody>
</table>

---
class: top left

# Select and rank significant harmonies (6 out of 16 selected)

![](images/global_harmony2.png)

---
class: top left

# Thank you

Slides: https://sayanigupta-ows2020.netlify.app/

Materials: https://github.com/Sayani07/OWS2020

Package (CRAN): v0.1.3

Package (Github) : https://github.com/Sayani07/gravitas

<br>

Slides created with <i> Rmarkdown, knitr, xaringan, xaringanthemer</i>