web-applications.Rmd

# Web application concepts {#web-applications}
In this chapter, we discuss the fundamental concepts underlying web applications, like the client-server model,
the HTTP protocol and web servers, showing how Shiny integrates that system and what the differences
are compared to the classic web standards. This chapter may significantly ease the understanding of Part 3. 

## The client-server model
A Shiny app is a __web application__, and like all web applications, 
it follows the __server-client__ model which consists of:

  - A __client__, which sends __requests__ to the server through the network.
  - A __server__ composed of hardware and software elements that treats the client request.
  - A __network__ inside which flow requests between the server and the client occur. It is done
  with the HyperText Transfer Protocol (HTTP).
  
Each time a client sends a request, it is processed by the server, which provides an answer and closes the connection before treating any other request. In practice, to get a web page, the client emits many requests, one to get the page and then one request per JS/CSS/image assets. As an example, try to run the following in the R console and open the developer tools:

```{r, eval=FALSE}
library(shiny)
ui <- fluidPage()
server <- function(input, output, session) {}
shinyApp(ui, server)
```

Under the network tab, we notice many files (if nothing is shown, reload the web browser tab), which actually correspond to all requests made by the client to the server, as seen in Figure \@ref(fig:shinyapp-requests). We also get the current answer status, 200 being the OK HTTP [status](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status), the size and the time needed to treat the request. Nowadays, there exists mechanisms like web browser caching to speed up the request treatment. Don't believe that each time you visit a Shiny app, all requests are answered by the server. Actually, most assets are recovered from the web browser cache, which takes significantly less time, although, sometimes misleading. I am sure you already found this situation when, after updating your Shiny app style, you still get the old design. Most of the time this is a caching issue and resetting Chrome's cache solves the problem.

```{r shinyapp-requests, echo=FALSE, fig.cap='Request flow between client and server at Shiny app start.', out.width='100%'}
knitr::include_graphics("images/survival-kit/shinyapp-requests.png")
```

## About HTTP requests {#web-applications-http}
If we inspect the first request from Figure \@ref(fig:shinyapp-requests), we obtain Figure \@ref(fig:http-request-details). An HTTP request is composed of:

  - A [__method__](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) that indicates the intentions. 
  We mostly use `GET` to ask for something or `POST`, to submit something. 
  - An __url__ corresponding to the path to the targeted element. Here, if nothing is specified in the path, the server will try to get the main HTML page, also called `index.html`. 

```{r http-request-details, echo=FALSE, fig.cap='Details about an HTTP request.', out.width='100%'}
knitr::include_graphics("images/survival-kit/http-request-details.png")
```

The HTTP protocol is unidirectional, that is, you may see it as a phone call during which you are only 
allowed one question, thereby terminating the call. 

`{httr}` [@R-httr] allows most of the HTTP request to come directly from R like:

```{r}
library(httr)
res <- GET("https://www.google.com")
```

## Structure of an URL
The url (uniform resource locator) defines the unique location of the content to access on the server. The general structure is
the [following](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL):

```
<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY>#<ANCHOR>
```

__PROTOCOL__ or (scheme) is the communication protocol, that is `HTTP` or `HTTPS` (encrypted HTTP). __HOST__ is an IP adress or a domain name, that may be bought. For instance `google.com` is an owned domain name. The __PORT__ indicates which program to use to access the specified resources (80 and 443 are the default HTTP and HTTPS values, respectively). __PATH__ is the location of the resource on the server. For instance, if you run the above basic Shiny app and enter `http://127.0.0.1:<PORT>/shared/jquery.min.js` (replace PORT by your own value), you'll see the jQuery.min.js code, which is actually needed by Shiny. __QUERY__ is the place to add extra parameters, following the `key/value` notation like `?key1=value1&...`. In a Shiny app, query parameters may be retrieved with `parseQueryString()` or altered by `updateQueryString()`. 

## Web app files structure {#web-app-file-structure}
There are substantial differences between Shiny and classic web applications regarding the project file structure. 
While web applications are composed of at least an `index.html` file, as well as optional 
pages and assets (extra CSS and JS files, images), we don't exactly find such a file structure in Shiny. 
A basic Shiny app folder is mostly composed of:

  - __app.R__ or __ui.R/server.R__
  - A __www__ folder containing assets like JS, CSS, images.
  
Where is the `index.html` file? It is actually created on the fly when you run `shinyApp()`. The detailed processes are 
mentioned later in this chapter. 
  
## Serving web apps
In order to expose our app to the rest of the world, we have to host it somewhere, that is a web server. 
A server actually corresponds to:

  - A __hardware__ layer, namely the machine or virtual machine able to run programs. Most of servers are running
  under Linux. It is actually pretty straightforward to set up your own server thanks to the many solutions like [digitalocean](https://www.digitalocean.com/) or the [Amazon Web Service](https://aws.amazon.com/), also known as AWS.
  - A __software__ layer, which are programs necessary to treat the client requests. You probably know [Apache](https://httpd.apache.org/) or [nginx](https://www.nginx.com/), which are the most common solutions.
  
How is a Shiny app served? In the Shiny context, we need software able to run R scripts, thereby preventing us from relying on classic hosting strategies. RStudio developed multiple solutions, the most famous is likely __shiny server__:

  - Shiny server [open source](https://rstudio.com/products/shiny/download-server/).
  - Shiny server [pro](https://rstudio.com/products/shiny-server-pro/).
  - [RStudio Connect](https://rstudio.com/products/connect/evaluation/).
  - [shinyapps.io](https://www.shinyapps.io/).
  
You can see Shiny server as an improved web server. Indeed, in addition to run Rmd documents (R markdown) or Shiny apps, it is able to interpret classic HTML files. An excellent guide developed by Dean Attali to set up your own server is available [here](https://deanattali.com/2015/05/09/setup-rstudio-shiny-server-digital-ocean/). 

Another noticeable difference between web servers and Shiny server is the running __port__, which defaults to `3838` for the latter (instead of the classic `80`), although entirely customizable through the configuration file. 

## About `{httpuv}`
In addition to the Shiny server layer, which is able to run R code and start any app on the server as a result of a user request, Shiny relies on `{httpuv}` [@R-httpuv] which fires a web server for each app directly from R, making it possible to handle HTTP requests but also the R and JS communication, which will be covered later in Chapter \@ref(shiny-intro). 

## Shiny app lifecycle
Whenever a user (client) accesses a Shiny app with his web browser, a series of events occurs (Figure \@ref(fig:shinyapp-lifecycle)):

  1. The client sends a HTTP `CONNECT` request to the server (Shiny server) containing the path 
  to the targeted app. 
  2. The Shiny server starts the targeted app with `runApp()`. 
  
Under the hood, `runApp()`:

  - Calls `shinyApp()`, which returns a Shiny app object composed of a server function and the UI.
    The UI has to be formatted to be a function returning an HTTP response, as requested by `{httpuv}`. Section \@ref(build-shiny-ui) explains this process in detail. 
  - Calls `startApp`, which creates HTTP and WebSocket (WS) handlers. WS handlers are responsible for controlling the WS behavior when the app starts, when a message is received from a client and when the app closes. WS are necessary communication between R and JS, as shown in Chapter \@ref(shiny-intro). 
  - Calls `startServer` from `{httpuv}`, which starts the HTTP server and opens the server WS connection. 
    
  3. If the R code does not contain errors, the server returns the Shiny UI HTML code to the client. 
  4. The HTML code is received and interpreted by the client web browser.
  5. The HTML page is rendered. It is an exact mirror of the initially provided `ui.R` code. 
  
```{r shinyapp-lifecycle, echo=FALSE, fig.cap='Shiny App lifecycle.', out.width='100%'}
knitr::include_graphics("images/survival-kit/shinyapp-lifecycle.png")
```

The returned HTML contains all the necessary JavaScript to subsequently open the client WS connection and start the dialog between R and JS. This will be discussed in Chapter \@ref(shiny-intro). 

### Building the UI {#build-shiny-ui}
As stated above in section \@ref(web-app-file-structure), the Shiny app file structure does not follow all the web development standards. Particularly, there is no `index.html` file. 

What definitely makes Shiny wonderful is the ability to only write R code to produce HTML. Although convenient for R users, there is a moment where all this R code has to become HTML, since web browsers are just not able to process R files. 

Shiny must provide a string containing the HTML code that will be later given to the `{httpuv}` server and displayed to the end user, if the request is successful. Moreover, it must be a valid HTML template, as shown in Chapter \@ref(simplest-html-template), which is not the case when you use top-level UI shiny function like `fluidPage()`:

```{r}
fluidPage(p())
```

In the above output, we miss the `<!DOCTYPE html>` indicating to the web browser that our
document is HTML and to load the appropriate interpreter. Additionally, `html`, `head` and `body` are not 
provided with `fluidPage()`.


How does Shiny create an appropriate HTML template? These steps heavily rely on `{htmltools}`, particularly the `renderDocument()` function. If this has not been documented in Chapter \@ref(htmltools-overview), it's mainly because it is, in theory, quite unlikely you'll ever use those functions, unless you try to develop another web framework for R, built on top of `{httpuv}`, like `{ambriorix}` or `{fiery}`. Another use case is `{argonR}`, which allows us to design Bootstrap 4 HTML templates, on top of the [argon](https://www.creative-tim.com/product/argon-design-system) design system. 

Under the hood, `shinyApp()` does many things, particularly creating a valid HTTP response template for `{httpuv}`, through the internal `shiny:::uiHttpHandler` function [^internal-shiny-ui]. The conversion from R to HTML is achieved by `shiny:::renderPage`. First, the provided UI R code is wrapped in a `tags$body()`, if not yet done. As a reminder `fluidPage` does not create a `body` tag, which is required to produce a valid HTML template. The result is given to `htmlTemplate()` to fill the following boilerplate, part of the Shiny package:

```{r, echo=FALSE, results='asis'}
html_code <- '<!DOCTYPE html>
<html{{ if (isTRUE(nzchar(lang))) 
  paste0(" lang=\"", lang, "\"") }}>
  <head>
  {{ headContent() }}
  </head>
  {{ body }}
</html>'
code_chunk_custom(html_code, "html")
```

If we assume that our UI is built as follows, applying `htmlTemplate()` on it yields:

```{r}
ui <- fluidPage(
  textInput("caption", "Caption", "Data Summary"),
  verbatimTextOutput("value")
)

ui <- htmlTemplate(
  system.file("template", "default.html", package = "shiny"), 
  lang = "en", 
  body = tags$body(ui), 
  document_ = TRUE
)
```

The output is shown below (body content is cropped for space reasons):

```{r, echo=FALSE, results='asis'}
html_code <- '<!DOCTYPE html>
<html lang="en">
  <head>
    <!-- HEAD_CONTENT -->
  </head>
  <body>
    <div class="container-fluid">
      <!-- Body content -->
    </div>
  </body>
</html>'
code_chunk_custom(html_code, "html")
```

You may wonder what `headContent()` does. It inserts the string `<!-- HEAD_CONTENT -->` inside the head tag so that
Shiny knows where to insert the dependencies, that is all mandatory JS and CSS assets. Then, all necessary dependencies like jQuery, Bootstrap and shiny css/javascript files (`shiny:::shinyDependencies`) are added in the UI head by `renderDocument()`. `renderDocument()` is a three steps process:

  - Convert all R Shiny tags to HTML with `renderTags()`. For each tag, `renderTags()` returns a list of
  four elements: the head content, any singletons, the list of dependencies and the HTML string. 
  - Treat the dependencies with `resolveDependencies()` to remove conflicts, as shown in Chapter \@ref(resolve-dependencies). 
  - Process the dependencies with `createWebDependency()`, which make sure each dependency can be served over HTTP.
  - Convert dependencies R code to HTML with `renderDependencies()` and insert it in the 
  template `head`, replacing the `<!-- HEAD_CONTENT -->` string. 

For instance, we call `renderTags()` on the Shiny `icon()`:

```{r, eval=FALSE}
library(htmltools)
res <- renderTags(icon("cogs"))
str(res)
```

```{r, echo=FALSE}
if (knitr::is_html_output()) {
  library(htmltools)
  res <- renderTags(icon("cogs"))
  str(res)
}
```

```{r, echo=FALSE, results='asis'}
code <- 'List of 4
 $ head        : \'html\' chr ""
  ..- attr(*, "html")= logi TRUE
 $ singletons  : chr(0) 
 $ dependencies:List of 1
  ..$ :List of 9
  .. ..$ name      : chr "font-awesome"
  .. ..$ version   : chr "5.13.0"
  .. ..$ src       :List of 1
  .. .. ..$ file: chr "<TRUNCATED>"
  .. ..$ meta      : NULL
  .. ..$ script    : NULL
  .. ..$ stylesheet: chr [1:2] "css/all.min.css" 
  "css/v4-shims.min.css"
  .. ..$ head      : NULL
  .. ..$ attachment: NULL
  .. ..$ all_files : logi TRUE
  .. ..- attr(*, "class")= chr "html_dependency"
 $ html        : \'html\' chr "<i class=\"fa fa-cogs\" 
 role=\"presentation\" aria-label=\"cogs icon\"></i>"
  ..- attr(*, "html")= logi TRUE'
exclude_from_html(code)
```

and then `renderDependencies()` on the tag dependency:

```{r, eval=FALSE}
renderDependencies(res$dependencies)
```

```{r, echo=FALSE}
if (knitr::is_html_output()) {
  renderDependencies(res$dependencies)
}
```

```{r, echo=FALSE, results='asis'}
code <- '<link href="TRUNCATED/fontawesome/css/all.min.css" 
rel="stylesheet" />
<link href="TRUNCATED/fontawesome/css/v4-shims.min.css" 
rel="stylesheet" />'
exclude_from_html(code)
```

Let's apply `renderDocument()` to our previous template:

```{r, message=FALSE, warning=FALSE}
html <- renderDocument(
  ui,
  deps = c(
    list(
      htmlDependency(
        "jquery", 
        "3.5.1", 
        c(href = "shared"), 
        script = "jquery.min.js"
      )
    ), 
    shiny:::shinyDependencies() # Shiny JS + CSS
  ),
  processDep = createWebDependency
)
```

The final HTML output is shown as follow (body content is cropped to save space). 
Look at the `head` tag where all dependencies are correctly inserted.

```{r, echo=FALSE, results='asis'}
html_code <- '<!DOCTYPE html>
<html lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; 
    charset=utf-8"/>
    <script type="application/shiny-singletons"></script>
    <script type="application/html-dependencies">jquery[3.5.1];
    shiny-css[1.6.0];shiny-javascript[1.6.0];
    bootstrap[3.4.1]</script>
    <script src="shared/jquery.min.js"></script>
    <link href="shared/shiny.min.css" rel="stylesheet" />
    <script src="shared/shiny.min.js"></script>
    <meta name="viewport" content="width=device-width, 
    initial-scale=1" />
    <link href="shared/bootstrap/css/bootstrap.min.css" 
    rel="stylesheet" />
    <link href="shared/bootstrap/accessibility/css/
    bootstrap-accessibility.min.css" rel="stylesheet" />
    <script src="shared/bootstrap/js/bootstrap.min.js"></script>
    <script src="shared/bootstrap/accessibility/js
    /bootstrap-accessibility.min.js"></script>
  </head>
  <body>
    <div class="container-fluid">
      <!-- Body content -->
    </div>
  </body>
</html>'
code_chunk_custom(html_code, "html")
```

The final step is to return an HTTP response containing the HTML string. As of `{shiny}` `1.6.0`, the `httpResponse` function is exported by default, and the returned content is exactly the same as showed above:

```{r, eval=FALSE}
httpResponse(
  status = 200,
  content = enc2utf8(paste(collapse = "\n", html))
)
```

```{r, echo=FALSE}
if (knitr::is_html_output()) {
  httpResponse(
    status = 200,
    content = enc2utf8(paste(collapse = "\n", html))
  )
}
```

```{r, echo=FALSE, results='asis'}
code <- '$status
[1] 200

$content_type
[1] "text/html; charset=UTF-8"

$content
[1] "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n  
  <meta http-equiv=\"Content-Type\" content=\"text/html; 
  charset=utf-8\"/>\n  
  <script type=\"application/shiny-singletons\"></script>\n  
  <script type=\"application/html-dependencies\">jquery[3.5.1];
  shiny-css[1.6.0];shiny-javascript[1.6.0];bootstrap[3.4.1]
  </script>\n
  <script src=\"shared/jquery.min.js\"></script>... TRUNCATED"

$headers
$headers$`X-UA-Compatible`
[1] "IE=edge,chrome=1"


attr(,"class")
[1] "httpResponse"'
exclude_from_html(code)
```


[^internal-shiny-ui]: The interested reader will have a look at the following [script](https://github.com/rstudio/shiny/blob/60db1e02b03d8e6fb146c9bb1bbfbce269231add/R/shinyui.R), which contains all
previously mentioned functions like `uiHttpHandler`.


### Serving HTML with `{httpuv}`
Once the UI is processed, Shiny makes it available to end users by leveraging `{httpuv}`, which provides tools to set up an HTTP server. The main function is `startServer`, which requires a __host__, __port__ and an __app__. If you run a Shiny app locally, the default host is `localhost` or `127.0.0.1`, and the port is randomly chosen by `shinyApp` or `runApp`, even though you may fix it. The most important element is the app, and `{httpuv}` expects a list of functions like:

  - `call`, to handle the client HTTP request and return the server HTTP response. Depending on the context,
  Shiny may return different responses like 403 (unauthorized), 404 (not found) or 200 (OK).
  - `onHeaders` if the request contains headers. For instance, this may be required for authentication.
  - `staticPaths` to serve assets, especially CSS or JS files. 
  
A valid `call` function template containing the previously processed HTML UI is defined below:

```{r, eval=FALSE}
app <- list()
app$call <- function(req) {
  list(
    status = 200L,
    headers = list(
      'Content-Type' = 'text/html'
    ),
    body = html
  )
}
```

We then invoke `startServer`:

```{r, eval=FALSE}
library(httpuv)
s <- startServer(
  "127.0.0.1",
  8080,
  app
)
```

Now, if we browse to `127.0.0.1:8080`, we see the text input. However, opening the HTML inspector shows many errors, most of them due to the fact that we forgot to serve static assets, all located in the `inst/www/shared` folder of the `{shiny}` package. Let's do it below by adding a `staticPaths` component to our app:

```{r, eval=FALSE}
s$stop() # stop the server before running it again!
app$staticPaths <- list(shared = system.file(
  package = "shiny", 
  "www", 
  "shared"
))
s <- startServer(
  "127.0.0.1",
  8080,
  app
)
```

The response may be inspecting directly from R (ideally within another R session) with an `{httr}` `GET` request:

```{r, eval=FALSE}
GET("http://127.0.0.1:8080")

## Response [http://127.0.0.1:8080]
##  Date: 2021-03-04 23:41
##  Status: 200
##  Content-Type: text/html
##  Size: 5 B
```

Keep in mind that Shiny does many more things to set up the server, and we just highlighted the most important steps.
The above code crashes since the HTML page returned to the client tries to connect to a server WS, that does not yet exist. 

## Summary
So far so good! You hopefully now better understand what a Shiny app is, how it is served and the differences between classic web apps.