chapter 5 finished

05-radar.Rmd

@@ -2,7 +2,7 @@
 
 Published in Journal of Medical Internet Research, 2019 (21);6, e12843
 
-Luz CF 1, Berends MS 1,2, Dik JWH 1, Lokate ML 1, Pulcini C 3,4, Glasner C 1, Sinha BNM 1
+Luz CF ^1^, Berends MS ^1,2^, Dik JWH ^1^, Lokate ML ^1^, Pulcini C ^3,4^, Glasner C ^1^, Sinha BNM ^1^
 
 1. University of Groningen, University Medical Center Groningen, Department of Medical Microbiology and Infection Prevention, Groningen, The Netherlands
 2. Certe Medical Diagnostics and Advice Foundation, Groningen, The Netherlands
@@ -17,53 +17,53 @@ Analysing process and outcome measures for all patients diagnosed with an infect
 
 ### Background
 
-With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as “a coherent set of actions which promote using antimicrobials responsibly” [1]. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.
+With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as “a coherent set of actions which promote using antimicrobials responsibly” ^[1]^. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.
 
-AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews [2-7]. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines [1]. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes [8].
+AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews ^[2-7]^. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines ^[1]^. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes ^[8]^.
 
 EHRs usually offer quick insights into useful infection management data on the individual patient level. However, easy access to analyse patient groups (e.g., stratified by departments or wards, specific antimicrobials, or diagnostic procedures used) is difficult to implement in daily practice. It is even more challenging to rapidly analyse larger patient populations (e.g., spread over multiple specialties) even though this information might be available. Nevertheless, this is vital for meaningful analysis, including possible confounders and pattern recognition across different populations. Moreover, when aggregated data are available, it is often not possible to trace individual patients, and analyses lack the ability to be further adjusted or stratified.
 
-AMS teams are multidisciplinary, and they act beyond the borders of single specialties [9]. They are usually understaffed, with limited data analysis support [10,11]. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.
+AMS teams are multidisciplinary, and they act beyond the borders of single specialties ^[9]^. They are usually understaffed, with limited data analysis support ^[10,11]^. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.
 
-Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives [1,9,12]. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.
+Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives ^[1,9,12]^. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.
 
-Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital [13]. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection [14]. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics [15].
+Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital ^[13]^. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection ^[14]^. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics ^[15]^.
 
-It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub [16], taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers [15].
+It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub ^[16]^, taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers ^[15]^.
 
-In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field [17-19]. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.
+In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field ^[17-19]^. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.
 
 ### Objectives
 
-We followed principles of open knowledge [20] to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app – Rapid analysis of diagnostic and antimicrobial patterns in R (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic R knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR’s analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.
+We followed principles of open knowledge ^[20]^ to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app – Rapid analysis of diagnostic and antimicrobial patterns in `R` (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic `R` knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR’s analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.
 
 ## Methods
 
-For the development of software in an open-source environment, we used the open-source programming language R in conjunction with RStudio version 1.1.463 (RStudio, Inc) [21], an open-source integrated desktop environment for R [22]. Both R and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for R [23]. Shiny allows R users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free [24], as well as on local or cloud-based servers or on personal computers.
+For the development of software in an open-source environment, we used the open-source programming language `R` in conjunction with RStudio version 1.1.463 (RStudio, Inc) ^[21]^, an open-source integrated desktop environment for `R` ^[22]^. Both `R` and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for `R` ^[23]^. Shiny allows `R` users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free ^[24]^, as well as on local or cloud-based servers or on personal computers.
 
-The functionality of R can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 [25], giving options to change, modify, and adapt RadaR to both personal and commercial users’ needs while requiring the need to document code changes [25].
+The functionality of `R` can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 ^[25]^, giving options to change, modify, and adapt RadaR to both personal and commercial users’ needs while requiring the need to document code changes ^[25]^.
 
 RadaR’s calculations and data aggregation are done reactively on the basis of the selection of the user. Single observations on the patient level build the basis for any calculation. RadaR uses common CSV files as input. A total of three different data sources are read in RadaR for admission, antimicrobial, and microbiological data, which are merged and transformed upon start. A patient number or study number is used as a unique identifier. All antimicrobial and microbiological data are checked to ascertain whether they fall in the interval of admission dates.
 
-<p class="tbl-caption">Table 1. Required R packages for RadaR.</p>
+<p class="tbl-caption">Table 1. Required `R` packages for RadaR.</p>
 
 ```{r tbl5-1}
-# insert_graphic("images/05-t01.jpg")
+insert_graphic("images/05-t01.jpg")
 ```
 
-The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham [26]. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the R package collection tidyverse for the preparation process [26,27]. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis [28-30]. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
+The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham ^[26]^. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the `R` package collection tidyverse for the preparation process ^[26,27]^. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis ^[28-30]^. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
 
 <p class="tbl-caption">Table 2. Input variables for RadaR.</p>
 
 ```{r tbl5-2}
-# insert_graphic("images/05-t02.jpg")
+insert_graphic("images/05-t02.jpg")
 ```
 
-The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham [26]. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the R package collection tidyverse for the preparation process [26,27]. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis [28-30]. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
+The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham ^[26]^. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the `R` package collection tidyverse for the preparation process ^[26,27]^. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis ^[28-30]^. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
 
-RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package [32]. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package [33]. Nonrandom variation in these run charts is tested using Anhøj’s rules [34].
+RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package ^[32]^. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package ^[33]^. Nonrandom variation in these run charts is tested using Anhøj’s rules ^[34]^.
 
-RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for R Shiny apps [35]. The entire source code of RadaR is freely accessible on GitHub [36]. We intend to integrate suggestions and feedback coming from its users and the R community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.
+RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for `R` Shiny apps ^[35]^. The entire source code of RadaR is freely accessible on GitHub ^[36]^. We intend to integrate suggestions and feedback coming from its users and the `R` community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.
 
 ## Results
 
@@ -78,13 +78,13 @@ RadaR is designed in the form of a web browser–based dashboard that most users
 <p class="tbl-caption">Table 3. Selection criteria in sidebar.</p>
 
 ```{r tbl5-3}
-# insert_graphic("images/05-t03.jpg")
+insert_graphic("images/05-t03.jpg")
 ```
 
 All output is based on the selection criteria defined by the user in the sidebar. Each new selection and any change need to be confirmed by clicking the confirm selection button (see Figure 1). Users can navigate among the different analysis panels by clicking the respective button.
 
 ```{r fig5-1, fig.cap = "Application design."}
-# insert_graphic("images/05-01.jpg")
+insert_graphic("images/05-01.jpg")
 ```
 
 Results are shown in bar charts, density plots, run charts, a bubble plot, and a Kaplan-Meier curve for LOS in hospital. Each panel further displays a table summarizing the respective data analyses. All output boxes and their content are described in Table 4. Most output boxes include modification options that can be identified by small gear icons (see Figure 1). These clickable icons allow for further specification of the generated plots and tables. Users can compare different groups (e.g., antimicrobial use by antimicrobial agent, resistance patterns per isolate, or LOS by specialty) or modify the plots (e.g., switch from count to proportion, change the chart type, or show or hide the legend). Plots and tables can be downloaded through download buttons as PNG files for plots and CSV, Excel, or PDF files for tables.
@@ -92,7 +92,7 @@ Results are shown in bar charts, density plots, run charts, a bubble plot, and a
 <p class="tbl-caption">Table 4. Output boxes for analysis results.</p>
 
 ```{r tbl5-4}
-# insert_graphic("images/05-t04.jpg")
+insert_graphic("images/05-t04.jpg")
 ```
 
 Finally, two datasets (antimicrobial/admission data and microbiological data) of the user-defined selection can be downloaded from the sidebar menu in a CSV-file format for further analysis (e.g., retrieving a list of patient numbers of the selected patient group).
@@ -118,12 +118,12 @@ A typical example workflow with RadaR comprises 6 steps (with examples from the
 
 For setting up RadaR in a new environment after data preparation, users only need to perform the following four steps:
 
-1.	Downloading R and RStudio [21,22], which are free to use and open-source software
-2.	Download or copy and paste RadaR’s source code [36] into three files in RStudio – global.R, server.R, and ui.R
-3.	In global.R, manually edit the paths for the prepared datasets to be imported into RadaR
-4.	Run the app in RStudio with the calling the function runApp() in the console or by clicking the green run app button. This will download and install the required R packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user’s operating system.
+1.	Downloading `R` and RStudio ^[21,22]^, which are free to use and open-source software
+2.	Download or copy and paste RadaR’s source code ^[36]^ into three files in RStudio – `global.R`, `server.R`, and `ui.R`
+3.	In `global.R`, manually edit the paths for the prepared datasets to be imported into RadaR
+4.	Run the app in RStudio with the calling the function `runApp()` in the console or by clicking the green run app button. This will download and install the required `R` packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user’s operating system.
 
-RadaR’s appearance has been customised using a cascading style sheets (CSS) script [37] that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (global.R, server.R, and ui.R) called www. We recommend RStudio’s project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR’s design by changing the underlying CSS script.
+RadaR’s appearance has been customised using a cascading style sheets (CSS) script ^[37]^ that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (`global.R`, `server.R`, and `ui.R`) called `www`. We recommend RStudio’s project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR’s design by changing the underlying CSS script.
 
 ## Discussion
 
@@ -131,11 +131,11 @@ RadaR’s appearance has been customised using a cascading style sheets (CSS) sc
 
 We have developed a web-based software app for rapid analysis of diagnostic and antimicrobial patterns that can support AMS teams to tailor their interventions. It has been designed to enhance communication of relevant findings while being easy to use. This also applies to users without extensive prior software skills, as it follows usual web browser user experiences. Moreover, it has been developed using open-source software. It is therefore free to use and accessible for download. In our experience, this system can be adapted to new settings within one day, when the required data (Table 2) are available.
 
-Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice [38]. However, it is difficult to compare functionalities of these tools because of their non–open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs [39,40].
+Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice ^[38]^. However, it is difficult to compare functionalities of these tools because of their non–open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs ^[39,40]^.
 
-The global nature of infections further calls to develop software tools applicable in resource-limited settings [41]. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis [15]. They highlight aspects that were also fundamental for the development of RadaR. First, R allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative R community on the web). Second, R is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.
+The global nature of infections further calls to develop software tools applicable in resource-limited settings ^[41]^. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis ^[15]^. They highlight aspects that were also fundamental for the development of RadaR. First, `R` allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative `R` community on the web). Second, `R` is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.
 
-Innovative approaches used in supporting infection management by leveraging EHRs are being investigated [17-19]. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches [17-19] either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards [42]. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.
+Innovative approaches used in supporting infection management by leveraging EHRs are being investigated ^[17-19]^. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches ^[17-19]^ either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards ^[42]^. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.
 
 ### Conclusions
 
@@ -143,7 +143,7 @@ RadaR demonstrates the feasibility of developing software tools for infection ma
 
 ## Acknowledgements {-}
 
-The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online R community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.
+The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online `R` community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.
 
 RadaR was developed as part of a project funded by the European Commission Horizon 2020 Framework Marie Skłodowska-Curie Actions (grant agreement number: 713660-PRONKJEWAIL-H2020-MSCA-COFUND-2015).
 

docs/a_new_instrument_for_microbial_epidemiology.tex

@@ -1824,7 +1824,7 @@ The funders had no role in study design, data collection and analysis, decision
 
 Published in Journal of Medical Internet Research, 2019 (21);6, e12843
 
-Luz CF 1, Berends MS 1,2, Dik JWH 1, Lokate ML 1, Pulcini C 3,4, Glasner C 1, Sinha BNM 1
+Luz CF \textsuperscript{1}, Berends MS \textsuperscript{1,2}, Dik JWH \textsuperscript{1}, Lokate ML \textsuperscript{1}, Pulcini C \textsuperscript{3,4}, Glasner C \textsuperscript{1}, Sinha BNM \textsuperscript{1}
 
 \begin{enumerate}
 \def\labelenumi{\arabic{enumi}.}
@@ -1851,47 +1851,63 @@ Analysing process and outcome measures for all patients diagnosed with an infect
 \hypertarget{background-1}{%
 \subsection{Background}\label{background-1}}
 
-With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as ``a coherent set of actions which promote using antimicrobials responsibly'' {[}1{]}. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.
+With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as ``a coherent set of actions which promote using antimicrobials responsibly'' \textsuperscript{{[}1{]}}. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.
 
-AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews {[}2-7{]}. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines {[}1{]}. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes {[}8{]}.
+AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews \textsuperscript{{[}2-7{]}}. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines \textsuperscript{{[}1{]}}. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes \textsuperscript{{[}8{]}}.
 
 EHRs usually offer quick insights into useful infection management data on the individual patient level. However, easy access to analyse patient groups (e.g., stratified by departments or wards, specific antimicrobials, or diagnostic procedures used) is difficult to implement in daily practice. It is even more challenging to rapidly analyse larger patient populations (e.g., spread over multiple specialties) even though this information might be available. Nevertheless, this is vital for meaningful analysis, including possible confounders and pattern recognition across different populations. Moreover, when aggregated data are available, it is often not possible to trace individual patients, and analyses lack the ability to be further adjusted or stratified.
 
-AMS teams are multidisciplinary, and they act beyond the borders of single specialties {[}9{]}. They are usually understaffed, with limited data analysis support {[}10,11{]}. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.
+AMS teams are multidisciplinary, and they act beyond the borders of single specialties \textsuperscript{{[}9{]}}. They are usually understaffed, with limited data analysis support \textsuperscript{{[}10,11{]}}. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.
 
-Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives {[}1,9,12{]}. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.
+Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives \textsuperscript{{[}1,9,12{]}}. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.
 
-Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital {[}13{]}. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection {[}14{]}. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics {[}15{]}.
+Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital \textsuperscript{{[}13{]}}. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection \textsuperscript{{[}14{]}}. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics \textsuperscript{{[}15{]}}.
 
-It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub {[}16{]}, taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers {[}15{]}.
+It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub \textsuperscript{{[}16{]}}, taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers \textsuperscript{{[}15{]}}.
 
-In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field {[}17-19{]}. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.
+In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field \textsuperscript{{[}17-19{]}}. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.
 
 \hypertarget{objectives}{%
 \subsection{Objectives}\label{objectives}}
 
-We followed principles of open knowledge {[}20{]} to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app -- Rapid analysis of diagnostic and antimicrobial patterns in R (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic R knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR's analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.
+We followed principles of open knowledge \textsuperscript{{[}20{]}} to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app -- Rapid analysis of diagnostic and antimicrobial patterns in \texttt{R} (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic \texttt{R} knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR's analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.
 
 \hypertarget{methods}{%
 \section{Methods}\label{methods}}
 
-For the development of software in an open-source environment, we used the open-source programming language R in conjunction with RStudio version 1.1.463 (RStudio, Inc) {[}21{]}, an open-source integrated desktop environment for R {[}22{]}. Both R and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for R {[}23{]}. Shiny allows R users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free {[}24{]}, as well as on local or cloud-based servers or on personal computers.
+For the development of software in an open-source environment, we used the open-source programming language \texttt{R} in conjunction with RStudio version 1.1.463 (RStudio, Inc) \textsuperscript{{[}21{]}}, an open-source integrated desktop environment for \texttt{R} \textsuperscript{{[}22{]}}. Both \texttt{R} and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for \texttt{R} \textsuperscript{{[}23{]}}. Shiny allows \texttt{R} users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free \textsuperscript{{[}24{]}}, as well as on local or cloud-based servers or on personal computers.
 
-The functionality of R can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 {[}25{]}, giving options to change, modify, and adapt RadaR to both personal and commercial users' needs while requiring the need to document code changes {[}25{]}.
+The functionality of \texttt{R} can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 \textsuperscript{{[}25{]}}, giving options to change, modify, and adapt RadaR to both personal and commercial users' needs while requiring the need to document code changes \textsuperscript{{[}25{]}}.
 
 RadaR's calculations and data aggregation are done reactively on the basis of the selection of the user. Single observations on the patient level build the basis for any calculation. RadaR uses common CSV files as input. A total of three different data sources are read in RadaR for admission, antimicrobial, and microbiological data, which are merged and transformed upon start. A patient number or study number is used as a unique identifier. All antimicrobial and microbiological data are checked to ascertain whether they fall in the interval of admission dates.
 
-Table 1. Required R packages for RadaR.
+Table 1. Required \texttt{R} packages for RadaR.
 
-The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of ``tidy data,'' as defined by Hadley Wickham {[}26{]}. Table 2 displays the set of variables underlying RadaR's functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the ``tidy data'' concept above and to the R package collection tidyverse for the preparation process {[}26,27{]}. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis {[}28-30{]}. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
+\begin{figure}
+
+{\centering \includegraphics[width=1\linewidth]{images/05-t01} 
+
+}
+
+\end{figure}
+
+The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of ``tidy data,'' as defined by Hadley Wickham \textsuperscript{{[}26{]}}. Table 2 displays the set of variables underlying RadaR's functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the ``tidy data'' concept above and to the \texttt{R} package collection tidyverse for the preparation process \textsuperscript{{[}26,27{]}}. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis \textsuperscript{{[}28-30{]}}. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
 
 Table 2. Input variables for RadaR.
 
-The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of ``tidy data,'' as defined by Hadley Wickham {[}26{]}. Table 2 displays the set of variables underlying RadaR's functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the ``tidy data'' concept above and to the R package collection tidyverse for the preparation process {[}26,27{]}. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis {[}28-30{]}. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
+\begin{figure}
 
-RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package {[}32{]}. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package {[}33{]}. Nonrandom variation in these run charts is tested using Anhøj's rules {[}34{]}.
+{\centering \includegraphics[width=1\linewidth]{images/05-t02} 
 
-RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for R Shiny apps {[}35{]}. The entire source code of RadaR is freely accessible on GitHub {[}36{]}. We intend to integrate suggestions and feedback coming from its users and the R community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.
+}
+
+\end{figure}
+
+The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of ``tidy data,'' as defined by Hadley Wickham \textsuperscript{{[}26{]}}. Table 2 displays the set of variables underlying RadaR's functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the ``tidy data'' concept above and to the \texttt{R} package collection tidyverse for the preparation process \textsuperscript{{[}26,27{]}}. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis \textsuperscript{{[}28-30{]}}. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.
+
+RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package \textsuperscript{{[}32{]}}. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package \textsuperscript{{[}33{]}}. Nonrandom variation in these run charts is tested using Anhøj's rules \textsuperscript{{[}34{]}}.
+
+RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for \texttt{R} Shiny apps \textsuperscript{{[}35{]}}. The entire source code of RadaR is freely accessible on GitHub \textsuperscript{{[}36{]}}. We intend to integrate suggestions and feedback coming from its users and the \texttt{R} community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.
 
 \hypertarget{results}{%
 \section{Results}\label{results}}
@@ -1908,12 +1924,37 @@ RadaR is designed in the form of a web browser--based dashboard that most users
 
 Table 3. Selection criteria in sidebar.
 
+\begin{figure}
+
+{\centering \includegraphics[width=1\linewidth]{images/05-t03} 
+
+}
+
+\end{figure}
+
 All output is based on the selection criteria defined by the user in the sidebar. Each new selection and any change need to be confirmed by clicking the confirm selection button (see Figure 1). Users can navigate among the different analysis panels by clicking the respective button.
 
+\begin{figure}
+
+{\centering \includegraphics[width=1\linewidth]{images/05-01} 
+
+}
+
+\caption{Application design.}\label{fig:fig5-1}
+\end{figure}
+
 Results are shown in bar charts, density plots, run charts, a bubble plot, and a Kaplan-Meier curve for LOS in hospital. Each panel further displays a table summarizing the respective data analyses. All output boxes and their content are described in Table 4. Most output boxes include modification options that can be identified by small gear icons (see Figure 1). These clickable icons allow for further specification of the generated plots and tables. Users can compare different groups (e.g., antimicrobial use by antimicrobial agent, resistance patterns per isolate, or LOS by specialty) or modify the plots (e.g., switch from count to proportion, change the chart type, or show or hide the legend). Plots and tables can be downloaded through download buttons as PNG files for plots and CSV, Excel, or PDF files for tables.
 
 Table 4. Output boxes for analysis results.
 
+\begin{figure}
+
+{\centering \includegraphics[width=1\linewidth]{images/05-t04} 
+
+}
+
+\end{figure}
+
 Finally, two datasets (antimicrobial/admission data and microbiological data) of the user-defined selection can be downloaded from the sidebar menu in a CSV-file format for further analysis (e.g., retrieving a list of patient numbers of the selected patient group).
 
 \hypertarget{development-process}{%
@@ -1954,16 +1995,16 @@ For setting up RadaR in a new environment after data preparation, users only nee
 \def\labelenumi{\arabic{enumi}.}
 \tightlist
 \item
-  Downloading R and RStudio {[}21,22{]}, which are free to use and open-source software
+  Downloading \texttt{R} and RStudio \textsuperscript{{[}21,22{]}}, which are free to use and open-source software
 \item
-  Download or copy and paste RadaR's source code {[}36{]} into three files in RStudio -- global.R, server.R, and ui.R
+  Download or copy and paste RadaR's source code \textsuperscript{{[}36{]}} into three files in RStudio -- \texttt{global.R}, \texttt{server.R}, and \texttt{ui.R}
 \item
-  In global.R, manually edit the paths for the prepared datasets to be imported into RadaR
+  In \texttt{global.R}, manually edit the paths for the prepared datasets to be imported into RadaR
 \item
-  Run the app in RStudio with the calling the function runApp() in the console or by clicking the green run app button. This will download and install the required R packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user's operating system.
+  Run the app in RStudio with the calling the function \texttt{runApp()} in the console or by clicking the green run app button. This will download and install the required \texttt{R} packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user's operating system.
 \end{enumerate}
 
-RadaR's appearance has been customised using a cascading style sheets (CSS) script {[}37{]} that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (global.R, server.R, and ui.R) called www. We recommend RStudio's project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR's design by changing the underlying CSS script.
+RadaR's appearance has been customised using a cascading style sheets (CSS) script \textsuperscript{{[}37{]}} that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (\texttt{global.R}, \texttt{server.R}, and \texttt{ui.R}) called \texttt{www}. We recommend RStudio's project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR's design by changing the underlying CSS script.
 
 \hypertarget{discussion-1}{%
 \section{Discussion}\label{discussion-1}}
@@ -1973,11 +2014,11 @@ RadaR's appearance has been customised using a cascading style sheets (CSS) scri
 
 We have developed a web-based software app for rapid analysis of diagnostic and antimicrobial patterns that can support AMS teams to tailor their interventions. It has been designed to enhance communication of relevant findings while being easy to use. This also applies to users without extensive prior software skills, as it follows usual web browser user experiences. Moreover, it has been developed using open-source software. It is therefore free to use and accessible for download. In our experience, this system can be adapted to new settings within one day, when the required data (Table 2) are available.
 
-Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice {[}38{]}. However, it is difficult to compare functionalities of these tools because of their non--open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs {[}39,40{]}.
+Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice \textsuperscript{{[}38{]}}. However, it is difficult to compare functionalities of these tools because of their non--open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs \textsuperscript{{[}39,40{]}}.
 
-The global nature of infections further calls to develop software tools applicable in resource-limited settings {[}41{]}. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis {[}15{]}. They highlight aspects that were also fundamental for the development of RadaR. First, R allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative R community on the web). Second, R is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.
+The global nature of infections further calls to develop software tools applicable in resource-limited settings \textsuperscript{{[}41{]}}. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis \textsuperscript{{[}15{]}}. They highlight aspects that were also fundamental for the development of RadaR. First, \texttt{R} allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative \texttt{R} community on the web). Second, \texttt{R} is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.
 
-Innovative approaches used in supporting infection management by leveraging EHRs are being investigated {[}17-19{]}. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches {[}17-19{]} either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards {[}42{]}. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.
+Innovative approaches used in supporting infection management by leveraging EHRs are being investigated \textsuperscript{{[}17-19{]}}. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches \textsuperscript{{[}17-19{]}} either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards \textsuperscript{{[}42{]}}. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.
 
 \hypertarget{conclusions}{%
 \subsection{Conclusions}\label{conclusions}}
@@ -1988,7 +2029,7 @@ RadaR demonstrates the feasibility of developing software tools for infection ma
 \section*{Acknowledgements}\label{acknowledgements-1}}
 \addcontentsline{toc}{section}{Acknowledgements}
 
-The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online R community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.
+The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online \texttt{R} community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.
 
 RadaR was developed as part of a project funded by the European Commission Horizon 2020 Framework Marie Skłodowska-Curie Actions (grant agreement number: 713660-PRONKJEWAIL-H2020-MSCA-COFUND-2015).
 

docs/ch05-radar.html

@@ -266,7 +266,7 @@ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warni
 <div id="ch05-radar" class="section level1" number="5">
 <h1><span class="header-section-number">5</span> Rapid Analysis of Diagnostic and Antimicrobial Patterns in R (RadaR): Interactive Open-Source Software App for Infection Management and Antimicrobial Stewardship</h1>
 <p>Published in Journal of Medical Internet Research, 2019 (21);6, e12843</p>
-<p>Luz CF 1, Berends MS 1,2, Dik JWH 1, Lokate ML 1, Pulcini C 3,4, Glasner C 1, Sinha BNM 1</p>
+<p>Luz CF <sup>1</sup>, Berends MS <sup>1,2</sup>, Dik JWH <sup>1</sup>, Lokate ML <sup>1</sup>, Pulcini C <sup>3,4</sup>, Glasner C <sup>1</sup>, Sinha BNM <sup>1</sup></p>
 <ol style="list-style-type: decimal">
 <li>University of Groningen, University Medical Center Groningen, Department of Medical Microbiology and Infection Prevention, Groningen, The Netherlands</li>
 <li>Certe Medical Diagnostics and Advice Foundation, Groningen, The Netherlands</li>
@@ -281,35 +281,37 @@ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warni
 <h2><span class="header-section-number">5.1</span> Introduction</h2>
 <div id="background-1" class="section level3" number="5.1.1">
 <h3><span class="header-section-number">5.1.1</span> Background</h3>
-<p>With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as “a coherent set of actions which promote using antimicrobials responsibly” [1]. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.</p>
-<p>AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews [2-7]. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines [1]. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes [8].</p>
+<p>With antimicrobial resistance (AMR) on the rise, efforts are being made worldwide to focus on the preservation of antimicrobials as a precious non-renewable resource. Infection management in the form of antimicrobial stewardship (AMS) programs has emerged as an effective solution to address this global health problem in hospitals. AMS programs are defined as “a coherent set of actions which promote using antimicrobials responsibly” <sup>[1]</sup>. Stewardship interventions and activities focus on individual patients (personalised medicine and consulting) as well as patient groups or clinical syndromes (guidelines, protocols, information technology infrastructure, and clinical decision support systems) while prioritizing improvement in quality of care and patient safety for any intervention. The appropriate use of antimicrobials based on accurate and timely diagnostics is integral for the successful management of infections. In doing so, the diagnostics contribute to efforts in minimizing AMR by optimizing the use of antimicrobials.</p>
+<p>AMS setups in hospitals are often heterogeneous, but audit and feedback to assess the goals are essential parts of most programs, and they are included in international guidelines and reviews <sup>[2-7]</sup>. Important data for AMS programs include, for example, days of therapy (DOT), daily defined doses (DDD), admission dates, length of stay (LOS), and adherence to local or national diagnostic, therapeutic, or infection management guidelines <sup>[1]</sup>. Clinical outcomes, quality of care, or consumption of hospital resources can be measured, for example, using mortality data or surrogate parameters such as LOS. The collection of these data is facilitated by electronic health records (EHRs) and administrative local databases. Notably, administrative data have also been shown to be a reliable source for assessing clinical outcomes <sup>[8]</sup>.</p>
 <p>EHRs usually offer quick insights into useful infection management data on the individual patient level. However, easy access to analyse patient groups (e.g., stratified by departments or wards, specific antimicrobials, or diagnostic procedures used) is difficult to implement in daily practice. It is even more challenging to rapidly analyse larger patient populations (e.g., spread over multiple specialties) even though this information might be available. Nevertheless, this is vital for meaningful analysis, including possible confounders and pattern recognition across different populations. Moreover, when aggregated data are available, it is often not possible to trace individual patients, and analyses lack the ability to be further adjusted or stratified.</p>
-<p>AMS teams are multidisciplinary, and they act beyond the borders of single specialties [9]. They are usually understaffed, with limited data analysis support [10,11]. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.</p>
-<p>Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives [1,9,12]. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.</p>
-<p>Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital [13]. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection [14]. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics [15].</p>
-<p>It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub [16], taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers [15].</p>
-<p>In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field [17-19]. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.</p>
+<p>AMS teams are multidisciplinary, and they act beyond the borders of single specialties <sup>[9]</sup>. They are usually understaffed, with limited data analysis support <sup>[10,11]</sup>. Therefore, they need user-friendly and time-saving data analysis resources, without the need for profound technical expertise once the system is set up. Aggregating and linking data of antimicrobial use, guideline adherence, and clinical outcomes at the institutional level can build the basis for important insights for these teams. These could be used to identify areas within hospitals that might benefit most from supportive AMS interventions (e.g., subspecialties with lower guideline adherence or unusual patterns of antimicrobial use). Moreover, feedback from these data could help physicians better understand their patient population as a whole; in addition, hospital administration could allocate resources in a more targeted fashion.</p>
+<p>Furthermore, aggregated data and simultaneous analysis of multiple areas (e.g., use of diagnostics and antimicrobials) present an extensive insight into large patient populations. This also enables the development of comprehensive and multidisciplinary approaches of infection management, combining diagnostic and therapeutic perspectives <sup>[1,9,12]</sup>. Unfortunately, these kinds of analyses still require substantial statistical knowledge and software skills, and it is time consuming when performed.</p>
+<p>Technology, data science, and software app development can bring solutions to complex data handling problems such as those described above. Software app development for medical and epidemiological (research) questions has found many important answers during recent years. For example, software apps at hospital emergency departments (EDs) in the form of a dashboard have been shown to improve efficiency and quality of care for patients requiring emergency admission to hospital <sup>[13]</sup>. These software apps are used to communicate clearly defined clinical problems, such as mortality ratio, number of cardiac arrests, or readmission rate to the EDs. This has led to a decreased LOS and mortality at the EDs. Others used similar approaches to rapidly and interactively display geographical locations of tuberculosis cases without the need of technical expertise improving the understanding of transmission and detection <sup>[14]</sup>. Furthermore, data-driven fields such as genomics are front runners in developing new, innovative software apps to handle large datasets, in close collaboration with bioinformatics <sup>[15]</sup>.</p>
+<p>It is important to note that all of these abovementioned software apps have been created in an open-source approach. This means that the underlying source code can be easily shared, easily modified, and freely distributed through open repositories, such as GitHub <sup>[16]</sup>, taking open-source software license obligations into account. This facilitates collaboration, quality control through code review, and easy adaptation to many different settings and information technology systems, and this supports the use of advanced data visualisation for users with minimal experience in programming and little or no budget for professional database engineers <sup>[15]</sup>.</p>
+<p>In the field of medical microbiology, different approaches have been described to interactively work with microbiological diagnostics data and EHRs: electronic antibiograms, centralised resistance analysis, EHR data mining, and clinical decision support systems for AMS are great examples for innovation in the field <sup>[17-19]</sup>. However, a full open-source approach for software apps working with combined antimicrobials use and diagnostic data of individual patients on the hospital level in the field of infection management is still lacking.</p>
 </div>
 <div id="objectives" class="section level3" number="5.1.2">
 <h3><span class="header-section-number">5.1.2</span> Objectives</h3>
-<p>We followed principles of open knowledge [20] to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app – Rapid analysis of diagnostic and antimicrobial patterns in R (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic R knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR’s analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.</p>
+<p>We followed principles of open knowledge <sup>[20]</sup> to address the need for an interactive, easy-to-use software app that allows users to investigate antimicrobial use, microbiological diagnostic use, and patient outcomes at an institutional (hospital) level. We developed an open-source, web-based software app – Rapid analysis of diagnostic and antimicrobial patterns in <code>R</code> (RadaR) that can be used for AMS and infection management. This free software app can be run on regular computers or implemented on local or web-based servers to be accessed through standard web browsers. The focus user group of this software app is health care professionals involved in AMS (e.g., infectious disease specialists, clinical microbiologists, and pharmacists). Although some technical expertise (basic <code>R</code> knowledge) is needed for installation and implementation, the use of RadaR follows usual web browser user experiences. RadaR enables rapid and reproducible data analysis without extensive previous analysis expertise in a graphically appealing way while being adaptable to different settings. RadaR’s analyses are based on datasets of individual patients. Therefore, aggregated results can also be stripped down, and additional patient features can be investigated. With this software app, we aim at supporting data-driven hospital insights and decision making for actors in the field of AMS in a free, transparent, and reproducible way.</p>
 </div>
 </div>
 <div id="methods" class="section level2" number="5.2">
 <h2><span class="header-section-number">5.2</span> Methods</h2>
-<p>For the development of software in an open-source environment, we used the open-source programming language R in conjunction with RStudio version 1.1.463 (RStudio, Inc) [21], an open-source integrated desktop environment for R [22]. Both R and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for R [23]. Shiny allows R users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free [24], as well as on local or cloud-based servers or on personal computers.</p>
-<p>The functionality of R can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 [25], giving options to change, modify, and adapt RadaR to both personal and commercial users’ needs while requiring the need to document code changes [25].</p>
+<p>For the development of software in an open-source environment, we used the open-source programming language <code>R</code> in conjunction with RStudio version 1.1.463 (RStudio, Inc) <sup>[21]</sup>, an open-source integrated desktop environment for <code>R</code> <sup>[22]</sup>. Both <code>R</code> and RStudio are free of charge, and they need to be installed for the development and implementation of RadaR. To build RadaR as a web-based software app, we used the Shiny package for <code>R</code> <sup>[23]</sup>. Shiny allows <code>R</code> users to build interactive web apps without extensive knowledge in web design and its programming languages. The web apps can be run and hosted on the web for free <sup>[24]</sup>, as well as on local or cloud-based servers or on personal computers.</p>
+<p>The functionality of <code>R</code> can be easily extended by installing additional packages. All packages used for the development of RadaR are listed in Table 1. RadaR is developed in an open-source environment and licensed under GNU General Public License v2.0 <sup>[25]</sup>, giving options to change, modify, and adapt RadaR to both personal and commercial users’ needs while requiring the need to document code changes <sup>[25]</sup>.</p>
 <p>RadaR’s calculations and data aggregation are done reactively on the basis of the selection of the user. Single observations on the patient level build the basis for any calculation. RadaR uses common CSV files as input. A total of three different data sources are read in RadaR for admission, antimicrobial, and microbiological data, which are merged and transformed upon start. A patient number or study number is used as a unique identifier. All antimicrobial and microbiological data are checked to ascertain whether they fall in the interval of admission dates.</p>
 <p class="tbl-caption">
-Table 1. Required R packages for RadaR.
+Table 1. Required <code>R</code> packages for RadaR.
 </p>
-<p>The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham [26]. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the R package collection tidyverse for the preparation process [26,27]. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis [28-30]. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.</p>
+<p><img src="images/05-t01.jpg" width="100%" style="display: block; margin: auto;" /></p>
+<p>The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham <sup>[26]</sup>. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the <code>R</code> package collection tidyverse for the preparation process <sup>[26,27]</sup>. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis <sup>[28-30]</sup>. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.</p>
 <p class="tbl-caption">
 Table 2. Input variables for RadaR.
 </p>
-<p>The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham [26]. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the R package collection tidyverse for the preparation process [26,27]. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis [28-30]. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.</p>
-<p>RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package [32]. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package [33]. Nonrandom variation in these run charts is tested using Anhøj’s rules [34].</p>
-<p>RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for R Shiny apps [35]. The entire source code of RadaR is freely accessible on GitHub [36]. We intend to integrate suggestions and feedback coming from its users and the R community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.</p>
+<p><img src="images/05-t02.jpg" width="100%" style="display: block; margin: auto;" /></p>
+<p>The input data should be structured in a dataset format, where each variable is one column and each observation is one row. This follows the concept of “tidy data,” as defined by Hadley Wickham <sup>[26]</sup>. Table 2 displays the set of variables underlying RadaR’s functionality. In our setting for the development of RadaR, these variables originated from three different data sources: administrative data from the hospital data warehouse, microbiological data from the laboratory information system, and antimicrobial prescription data from the computerised prescriber order entry system. The data preparation and cleaning process are very specific for each data source, dependent on local data standards, and difficult to generalise. Therefore, Table 2 represents the final variables and formats for the analysis and use with RadaR, referring to the “tidy data” concept above and to the <code>R</code> package collection tidyverse for the preparation process <sup>[26,27]</sup>. Additional variables are calculated and transformed using the packages lubridate and zoo for time points and intervals, and AMR for antimicrobial (group) names, microbial isolate names, first isolate identification, and resistance analysis <sup>[28-30]</sup>. Microbiological resistance is calculated per antimicrobial substance or as co-resistance if more than one substance is selected.</p>
+<p>RadaR can be used for graphical exploratory data analysis. Differences in LOS are displayed by a Kaplan-Meier curve in conjunction with a log-rank test, using the survminer package <sup>[32]</sup>. Time trends for number of admissions, antimicrobial consumption, and resistance counts per year, quarter, or month, are visualised in run charts using the qicharts2 package <sup>[33]</sup>. Nonrandom variation in these run charts is tested using Anhøj’s rules <sup>[34]</sup>.</p>
+<p>RadaR has been developed in macOS High Sierra (1.4 GHz, 4 GB RAM), and it was successfully tested in Windows 7 (3.2 GHz, 8 GB RAM) and Linux (Ubuntu 16.04.4 LTS, 3.4 GHz, 12 GB RAM). A running example version has been deployed to shinyapps.io, a publicly available web hosting service for <code>R</code> Shiny apps <sup>[35]</sup>. The entire source code of RadaR is freely accessible on GitHub <sup>[36]</sup>. We intend to integrate suggestions and feedback coming from its users and the <code>R</code> community. RadaR was developed using data of patients admitted to the University Medical Center Groningen, Groningen, the Netherlands. Data were collected retrospectively, and permission was granted by the ethical committee (METc 2014/530). RadaR can be used locally in protected environments or hosted on the web, provided appropriate measures have been taken to guarantee data protection, depending on national regulations.</p>
 </div>
 <div id="results" class="section level2" number="5.3">
 <h2><span class="header-section-number">5.3</span> Results</h2>
@@ -323,11 +325,19 @@ Table 2. Input variables for RadaR.
 <p class="tbl-caption">
 Table 3. Selection criteria in sidebar.
 </p>
+<p><img src="images/05-t03.jpg" width="100%" style="display: block; margin: auto;" /></p>
 <p>All output is based on the selection criteria defined by the user in the sidebar. Each new selection and any change need to be confirmed by clicking the confirm selection button (see Figure 1). Users can navigate among the different analysis panels by clicking the respective button.</p>
+<div class="figure" style="text-align: center"><span id="fig:fig5-1"></span>
+<img src="images/05-01.jpg" alt="Application design." width="100%" />
+<p class="caption">
+Figure 5.1: Application design.
+</p>
+</div>
 <p>Results are shown in bar charts, density plots, run charts, a bubble plot, and a Kaplan-Meier curve for LOS in hospital. Each panel further displays a table summarizing the respective data analyses. All output boxes and their content are described in Table 4. Most output boxes include modification options that can be identified by small gear icons (see Figure 1). These clickable icons allow for further specification of the generated plots and tables. Users can compare different groups (e.g., antimicrobial use by antimicrobial agent, resistance patterns per isolate, or LOS by specialty) or modify the plots (e.g., switch from count to proportion, change the chart type, or show or hide the legend). Plots and tables can be downloaded through download buttons as PNG files for plots and CSV, Excel, or PDF files for tables.</p>
 <p class="tbl-caption">
 Table 4. Output boxes for analysis results.
 </p>
+<p><img src="images/05-t04.jpg" width="100%" style="display: block; margin: auto;" /></p>
 <p>Finally, two datasets (antimicrobial/admission data and microbiological data) of the user-defined selection can be downloaded from the sidebar menu in a CSV-file format for further analysis (e.g., retrieving a list of patient numbers of the selected patient group).</p>
 </div>
 <div id="development-process" class="section level3" number="5.3.3">
@@ -351,12 +361,12 @@ Table 4. Output boxes for analysis results.
 <h3><span class="header-section-number">5.3.5</span> Customisation</h3>
 <p>For setting up RadaR in a new environment after data preparation, users only need to perform the following four steps:</p>
 <ol style="list-style-type: decimal">
-<li>Downloading R and RStudio [21,22], which are free to use and open-source software</li>
-<li>Download or copy and paste RadaR’s source code [36] into three files in RStudio – global.R, server.R, and ui.R</li>
-<li>In global.R, manually edit the paths for the prepared datasets to be imported into RadaR</li>
-<li>Run the app in RStudio with the calling the function runApp() in the console or by clicking the green run app button. This will download and install the required R packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user’s operating system.</li>
+<li>Downloading <code>R</code> and RStudio <sup>[21,22]</sup>, which are free to use and open-source software</li>
+<li>Download or copy and paste RadaR’s source code <sup>[36]</sup> into three files in RStudio – <code>global.R</code>, <code>server.R</code>, and <code>ui.R</code></li>
+<li>In <code>global.R</code>, manually edit the paths for the prepared datasets to be imported into RadaR</li>
+<li>Run the app in RStudio with the calling the function <code>runApp()</code> in the console or by clicking the green run app button. This will download and install the required <code>R</code> packages needed for the app if they have not been installed previously, and this will create the final dataset for analysis. The RadaR interface will open in the RStudio viewer pane or in a new window of the standard browser of the user’s operating system.</li>
 </ol>
-<p>RadaR’s appearance has been customised using a cascading style sheets (CSS) script [37] that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (global.R, server.R, and ui.R) called www. We recommend RStudio’s project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR’s design by changing the underlying CSS script.</p>
+<p>RadaR’s appearance has been customised using a cascading style sheets (CSS) script <sup>[37]</sup> that is loaded into the app upon its start. This script needs to be saved into a subdirectory of the directory of the three main files (<code>global.R</code>, <code>server.R</code>, and <code>ui.R</code>) called <code>www</code>. We recommend RStudio’s project function to create a single project for RadaR and to store all information in this project directory. Users with experience in using CSS can fully alter RadaR’s design by changing the underlying CSS script.</p>
 </div>
 </div>
 <div id="discussion-1" class="section level2" number="5.4">
@@ -364,9 +374,9 @@ Table 4. Output boxes for analysis results.
 <div id="principal-findings" class="section level3" number="5.4.1">
 <h3><span class="header-section-number">5.4.1</span> Principal Findings</h3>
 <p>We have developed a web-based software app for rapid analysis of diagnostic and antimicrobial patterns that can support AMS teams to tailor their interventions. It has been designed to enhance communication of relevant findings while being easy to use. This also applies to users without extensive prior software skills, as it follows usual web browser user experiences. Moreover, it has been developed using open-source software. It is therefore free to use and accessible for download. In our experience, this system can be adapted to new settings within one day, when the required data (Table 2) are available.</p>
-<p>Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice [38]. However, it is difficult to compare functionalities of these tools because of their non–open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs [39,40].</p>
-<p>The global nature of infections further calls to develop software tools applicable in resource-limited settings [41]. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis [15]. They highlight aspects that were also fundamental for the development of RadaR. First, R allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative R community on the web). Second, R is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.</p>
-<p>Innovative approaches used in supporting infection management by leveraging EHRs are being investigated [17-19]. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches [17-19] either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards [42]. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.</p>
+<p>Commercial software for infection management is available (e.g., Epic Antimicrobial Stewardship Module, TREAT Steward). These offer extensive options for filtering, analysing, and visualizing EHRs with real-time connections to hospital data infrastructures and have been shown to be useful in clinical practice <sup>[38]</sup>. However, it is difficult to compare functionalities of these tools because of their non–open-source nature. This fact, along with the required budget to purchase the software, drastically limits their use. We are convinced that transparent software development can support the adoption of data-driven developments while enhancing optimal quality of care and patient safety, which is crucial in the light of new data-driven developments of using EHRs <sup>[39,40]</sup>.</p>
+<p>The global nature of infections further calls to develop software tools applicable in resource-limited settings <sup>[41]</sup>. Open-source approaches for data analysis, such as RadaR, have advantages over traditional methods, such as Excel or SPSS. Hughes et al described those in their report of a software app for RNA-sequencing data analysis <sup>[15]</sup>. They highlight aspects that were also fundamental for the development of RadaR. First, <code>R</code> allows transparent, reproducible, and sustainable data analysis through scripts that can easily be shared and changed. This can build the basis for collaboration, and this enforces the spirit of open science (also through the strong collaborative <code>R</code> community on the web). Second, <code>R</code> is open source and free to use; therefore, it also enables use in resource-limited settings. Finally, Shiny empowers users to interact with the data, making even very large datasets quickly interpretable.</p>
+<p>Innovative approaches used in supporting infection management by leveraging EHRs are being investigated <sup>[17-19]</sup>. Reporting on AMR, antimicrobial use, and hospital infections (e.g., for quality assurance) is well established, but it is important to integrate these data sources in an approach that allows detailed filtering options on all input. Merely looking at antimicrobial use alone or comparing aggregated results (e.g., total amount of a specific antimicrobial substance per hospital correlated with the total count of a resistant isolate) will result in loss of information or even misleading interpretation. Detailed data and calculations on the basis of each individual patient are crucial to draw informed conclusions. Unfortunately, the abovementioned infection management approaches <sup>[17-19]</sup> either depend on additional commercial software for data visualisation or the source code is not openly available. We want to encourage others to turn toward available open-source software solutions, such as R, for an increased potential of collaboration and transparency. However, their strength is the connection to real-time data flows. This enables the prospective use and increases their usability for daily clinical practice. RadaR is currently still limited to retrospective data analysis because of a changing hospital data infrastructure in our setting. Technically, it is feasible to connect R-based software apps such as RadaR to real-time hospital data infrastructures running with clinical data standards <sup>[42]</sup>. For a start, access to static data extraction is often easier and faster to achieve. RadaR can be used to advocate the use of data visualisation tools and improved accessibility of hospital data sources. Until connection to real-time hospital data is established, RadaR can support users as a stand-alone option for retrospective data analysis in infection management. Next steps will involve testing in multiple settings and forming a user and research group to continue and expand the use of open-source technology and open science principles in infection management.</p>
 </div>
 <div id="conclusions" class="section level3" number="5.4.2">
 <h3><span class="header-section-number">5.4.2</span> Conclusions</h3>
@@ -375,7 +385,7 @@ Table 4. Output boxes for analysis results.
 </div>
 <div id="acknowledgements-1" class="section level2 unnumbered">
 <h2>Acknowledgements</h2>
-<p>The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online R community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.</p>
+<p>The authors would like to thank the ESGAP executive committee for supporting the evaluation of RadaR in the ESGAP study group and all its members for their valuable input, suggestions, and comments. Furthermore, the authors wish to thank Igor van der Weide, Jan Arends, and Prashant Nannan Panday for their great support in obtaining required data at our institution that built the basis for the development of RadaR. The authors also thank the online <code>R</code> community as well as the valuable comments, suggestions, and input from reviewers that they have received to improve RadaR.</p>
 <p>RadaR was developed as part of a project funded by the European Commission Horizon 2020 Framework Marie Skłodowska-Curie Actions (grant agreement number: 713660-PRONKJEWAIL-H2020-MSCA-COFUND-2015).</p>
 </div>
 <div id="conflicts-of-interests" class="section level2 unnumbered">

docs/reference-keys.txt

@@ -11,6 +11,7 @@ fig:fig2-5
 fig:fig3-1
 fig:fig3-2
 fig:fig4-1
+fig:fig5-1
 ch01-introduction
 microbial-epidemiology
 antimicrobial-resistance-in-microorganisms

style.css

@@ -69,10 +69,9 @@ li.chapter b:after{
 .book .book-body .section * {
   color: black;
 }
-.book-body p code {
+.book-body p code,
+.book-body li code{
   font-size: 1.3rem !important;
-  padding-left: 0.1em;
-  padding-right: 0.1em;
   white-space: unset;
 }
 .book-body p code:before,