---
title: "Subsetting Data in R"
author: "Data Wrangling in R"
output:
  ioslides_presentation:
    css: ../styles/styles.css
    widescreen: yes
  beamer_presentation: default
---

```{r, include = FALSE}
library(knitr)
opts_chunk$set(comment = "")
suppressPackageStartupMessages(library(dplyr))
library(dplyr)
library(ggplot2)
library(here)
```

## Overview

```{css, echo=FALSE}
.watch-out {
  #background-color: lightpink;
  border: 3px solid red;
  font-weight: bold;
}
# from: https://bookdown.org/yihui/rmarkdown-cookbook/chunk-styling.html

#Then just use class.source="watch-out" in the chunk parameters
```



We showed different ways to read data into R using:  

```{r, eval = FALSE}
readr::read_csv()
readr::read_delim()
readxl::read_excel()
```

In this module, we will show you how select rows and columns of datasets. 

## Setup


We will be using the $\color{red}{\text{dplyr}}$  package in the tidyverse.

Here are several resources on how to use `dplyr`: 

* https://dplyr.tidyverse.org/
* https://r4ds.had.co.nz/
* https://cran.rstudio.com/web/packages/dplyr/vignettes/dplyr.html
* https://stat545.com/dplyr-intro.html

The `dplyr` package also interfaces well with tibbles.

## Dataset

We will be using the `diamonds` dataset in the `ggplot2` package as an example (so make sure you initiate the `ggplot2` package if you are following along on your own). 

```{r}
head(diamonds)
```

## Selecting a single column of a `data.frame`: 

To grab just the values from a single column, you would use the `pull` function. The output will be a vector (and not a `tibble`). 

Since this is a long vector we will just show the first 6 values using the `head` function around the output of the `pull` function.

```{r}
head(pull(diamonds,carat))
```

## Using the `pipe` (comes with `dplyr`):

That was a lot of typing and nested functions, which can be confusing.  The pipe `%>%` or (you might see also this pipe `|>`) makes things such as this much more readable.  It reads left side "pipes" into right side.  RStudio `CMD/Ctrl + Shift + M` shortcut. 

## Using the `pipe` (comes with `dplyr`):

Pipe `diamonds` into `select`, then pipe that into `pull`, and then show the `head`:

```{r}
diamonds %>% pull(carat) %>% head()
```


## Selecting a single column of a `data.frame`:

The `pull` function is equivalent to using the `$` method (in base R). 

Note that base R and tidyverse don't always play nice together. 

```{r}
head(pull(diamonds, carat))
head(diamonds$carat)
```

Note this does *not* return a `tibble` (or `data.frame`) but rather a vector. 

## Selecting a single column of a `data.frame`: 

The `select` function extracts one or more columns from a `tibble` or `data.frame` and returns a `tibble` (not a vector).

```{r}
select(diamonds, carat)
```


## Selecting multiple columns of a `data.frame`: 

The `select` command from `dplyr` is very flexible. You just need to list all columns you want to extract separated by commas.  You can use this as a way to just keep the columns you want for example.

```{r}
select(diamonds, carat, depth)
```



## See the Select "helpers"

Type `tidyselect::` to see functions available.

```{r, echo = FALSE, fig.align = 'center', out.width="70%"}
knitr::include_graphics("https://github.com/SISBID/Data-Wrangling/blob/gh-pages/images/tidyselect.png?raw=true")
```

Here are a few:

```{r, eval = FALSE}
last_col()
ends_with()
starts_with()
contains() # search for a pattern
everything()
```

## Tidyselect helpers

For example, we can take all columns that start with a "c":

```{r}
diamonds %>% select(starts_with("c"))
```

## Tidyselect helpers

Or we can take all columns that end with an "e":

```{r}
diamonds %>% select(ends_with("e"))
```

## Tidyselect helpers

We are going to cover "fancier" ways of matching column names (and strings more generally) in the data cleaning lecture. 

## Subset rows of a `data.frame`:

The command in `dplyr` for subsetting rows is `filter`.  Try `?filter`.

The easiest way to filter is by testing whether numeric observations are greater than or less than some cutoff:

```{r}
filter(diamonds, depth > 60)
```

Note, no subsetting is necessary.  R "knows" `depth` refers to a column of `diamonds`.

## Subset using pipes:

You can also using piping here:

```{r}
diamonds %>% filter(depth > 60)
```

## Subset rows with multiple required conditions:

You can combine filtering on multiple columns by separating the filter arguments with `&` (or commas - but `&` is more clear)

```{r}
diamonds %>% filter(depth > 60 & table > 60 & price > 2775)
```

## Subset rows for a character condition:

You can also filter character strings by a single value or category. Here we need quotes around character strings. 

```{r}
diamonds %>% filter(color == "I" & 
            clarity == "SI2" & cut == "Premium")
```

## Subset to match a vector of possibilities:

Sometimes you want to be able to filter on matching several values or categories. The `%in%` operator is useful here:

```{r}
diamonds %>% filter(clarity %in% c("SI1", "SI2"))
```

## Subset rows using conditions for numeric and character variables:

You can mix and match filtering on numeric and categorical/character columns in the same `filter()` command:

```{r}
diamonds %>% filter(clarity %in% c("SI1", "SI2") &
                    cut == "Premium" & price > 3000)
```

## Note about quotes and numbers

R will interpret quotes around numbers as the characters themselves and not their numeric meaning. Thus it's generally best to avoid quotes around numeric unless it is not being treated as a numeric value - for example levels or grades.

```{r, eval = FALSE}
diamonds %>% filter(price > 3001) #This works
diamonds %>% filter(price > "3001") # This does not

diamonds %>% filter(price == 3001) # This works
diamonds %>% filter(price == "3001") # this works 
```


## Logic for subsetting rows of a `data.frame`:

Other useful logical tests:

`&` : AND

`|` : OR

`<=` : less than or equals

`>=` : greater than or equals

`!=` : not equals


## Subset rows with or logic:

The OR operator (`|`) is more permissive than the AND operator:

```{r}
diamonds %>% filter(depth > 60 | table > 60 | price > 2775)
```

## OR is like `%in%`:

The OR operator (`|`) can be a substitute for `%in%` (although it might take more typing):

```{r}
diamonds %>% filter(clarity =="SI1" | clarity == "SI2") %>% head(2)
diamonds %>% filter(clarity %in% c("SI1", "SI2")) %>% head(2)
```

## Combining `filter` and `select`:

You can combine `filter` and `select` to subset the rows and columns, respectively, of a `data.frame`:

```{r}
diamonds %>% 
  filter(clarity == "SI2") %>% 
  select(starts_with("c"))
```

## Combining `filter` and `select`- Order matters!:

The order of these functions matters though, since you can remove columns that you might want to filter on. 

```{r,eval=FALSE}
diamonds %>% 
  select(starts_with("c")) %>%
  filter(table > 60))
```

This will result in an error because the table column is now gone after the `select()` function!

# Fancier filtering

## Combining tidyselect helpers with regular selection

```{r}
head(diamonds, 2)
diamonds %>% select(price, starts_with("c"))
```
## Multiple tidyselect functions

Follows OR logic.

```{r}
diamonds %>% select(starts_with("c"), ends_with("e"))

```

## Multiple patterns with tidyselect

Need to combine the patterns with the `c()` function.

```{r}
diamonds %>% select(starts_with(c("c", "p")))

```

## Common error for filter or select

If you try to filter or select for a column that does not exist it will not work:
 
 - misspelled column name 
 - column that was already removed 
 
## Always good to check each step!

Did the filter work the way you expected? Did the dimensions change?

Don't use quotes on column names - nothing will get filtered!

```{r, echo = FALSE, fig.align='center'}
url = "https://media.giphy.com/media/5b5OU7aUekfdSAER5I/giphy.gif"
knitr::include_graphics(url)
```
Source: https://media.giphy.com/media/5b5OU7aUekfdSAER5I/giphy.gif

## Summary

- `pull()` can help us see a vector version of our variables - we can "pull" out the data from a dataframe
- The pipe (`%>%`) can help us to do sequential steps
- `select()` makes a smaller table of just selected variables
- tidyselect functions can help us select specific columns: `contains()`, `ends_with()`, `starts_with()`
- `filter` can remove rows based on conditions
- `==` is needed to filter for rows that are "exactly equal" to a value
- `!=` does the opposite


## Summary Continued

- `%in%` enables us to do multiple `==` conditions such as `%in% c(1,2,3)`
- `|` is for or logic and `&` is for and logic when combining filter conditions together
- Always check that you filtered for what you think you did!

## Lab

[Link to Lab](http://sisbid.github.io/Data-Wrangling/05_Data_Subsetting/lab/data-subsetting-lab-part1.Rmd)