AMR/vignettes/datasets.Rmd

---
title: "Data sets for download / own use"
date: '`r format(Sys.Date(), "%d %B %Y")`'
output: 
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 1
vignette: >
  %\VignetteIndexEntry{Data sets for download / own use}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  chunk_output_type: console
---

```{r setup, include = FALSE, results = "markup"}
knitr::opts_chunk$set(
  warning = FALSE,
  collapse = TRUE,
  comment = "#",
  fig.width = 7.5,
  fig.height = 5
)

library(AMR)
library(dplyr)

options(knitr.kable.NA = "")

structure_txt <- function(dataset) {
  paste0(
    "A data set with ",
    format(nrow(dataset), big.mark = " "), " rows and ",
    ncol(dataset), " columns, containing the following column names:  \n",
    AMR:::vector_or(colnames(dataset), quotes = "*", last_sep = " and ", sort = FALSE), "."
  )
}

download_txt <- function(filename) {
  msg <- paste0(
    "It was last updated on ",
    trimws(format(file.mtime(paste0("../data/", filename, ".rda")), "%e %B %Y %H:%M:%S %Z", tz = "UTC")),
    ". Find more info about the structure of this data set [here](https://msberends.github.io/AMR/reference/", ifelse(filename == "antivirals", "antibiotics", filename), ".html).\n"
  )
  github_base <- "https://github.com/msberends/AMR/raw/main/data-raw/"
  filename <- paste0("../data-raw/", filename)
  rds <- paste0(filename, ".rds")
  txt <- paste0(filename, ".txt")
  excel <- paste0(filename, ".xlsx")
  feather <- paste0(filename, ".feather")
  parquet <- paste0(filename, ".parquet")
  xpt <- paste0(filename, ".xpt")
  spss <- paste0(filename, ".sav")
  stata <- paste0(filename, ".dta")
  create_txt <- function(filename, type, software, exists) {
    if (isTRUE(exists)) {
      paste0(
        "* Download as [", software, "](", github_base, filename, ") (",
        AMR:::formatted_filesize(filename), ")  \n"
      )
    } else {
      paste0("* *(unavailable as ", software, ")*\n")
    }
  }

  if (any(
    file.exists(rds),
    file.exists(txt),
    file.exists(excel),
    file.exists(feather),
    file.exists(parquet),
    file.exists(xpt),
    file.exists(spss),
    file.exists(stata)
  )) {
    msg <- c(
      msg, "\n**Direct download links:**\n\n",
      create_txt(rds, "rds", "original R Data Structure (RDS) file", file.exists(rds)),
      create_txt(txt, "txt", "tab-separated text file", file.exists(txt)),
      create_txt(excel, "xlsx", "Microsoft Excel workbook", file.exists(excel)),
      create_txt(feather, "feather", "Apache Feather file", file.exists(feather)),
      create_txt(parquet, "parquet", "Apache Parquet file", file.exists(parquet)),
      create_txt(xpt, "xpt", "SAS transport (XPT) file", file.exists(xpt)),
      create_txt(spss, "sav", "IBM SPSS Statistics data file", file.exists(spss)),
      create_txt(stata, "dta", "Stata DTA file", file.exists(stata))
    )
  }
  paste0(msg, collapse = "")
}

print_df <- function(x, rows = 6) {
  x %>%
    as.data.frame(stringsAsFactors = FALSE) %>%
    head(n = rows) %>%
    mutate_all(function(x) {
      if (is.list(x)) {
        sapply(x, function(y) {
          if (length(y) > 3) {
            paste0(paste(y[1:3], collapse = ", "), ", ...")
          } else if (length(y) == 0 || all(is.na(y))) {
            ""
          } else {
            paste(y, collapse = ", ")
          }
        })
      } else {
        x
      }
    }) %>%
    knitr::kable(align = "c")
}
```

All reference data (about microorganisms, antibiotics, SIR interpretation, EUCAST rules, etc.) in this `AMR` package are reliable, up-to-date and freely available. We continually export our data sets to formats for use in R, MS Excel, Apache Feather, Apache Parquet, SPSS, SAS, and Stata. We also provide tab-separated text files that are machine-readable and suitable for input in any software program, such as laboratory information systems. 

On this page, we explain how to download them and how the structure of the data sets look like. 

## `microorganisms`: Full Microbial Taxonomy

`r structure_txt(microorganisms)`

This data set is in R available as `microorganisms`, after you load the `AMR` package.

`r download_txt("microorganisms")`

**NOTE: The exported files for SAS, SPSS and Stata contain only the first 50 SNOMED codes per record, as their file size would otherwise exceed 100 MB; the file size limit of GitHub.** Their file structures and compression techniques are very inefficient. Advice? Use R instead. It's free and much better in many ways.

The tab-separated text file and Microsoft Excel workbook both contain all SNOMED codes as comma separated values.

### Source

This data set contains the full microbial taxonomy of `r AMR:::nr2char(length(unique(AMR::microorganisms$kingdom[!AMR::microorganisms$kingdom %like% "unknown"])))` kingdoms from the List of Prokaryotic names with Standing in Nomenclature (LPSN) and the Global Biodiversity Information Facility (GBIF):

* `r AMR:::TAXONOMY_VERSION$LPSN$citation` Accessed from <`r AMR:::TAXONOMY_VERSION$LPSN$url`> on `r AMR:::documentation_date(AMR:::TAXONOMY_VERSION$LPSN$accessed_date)`.
* `r AMR:::TAXONOMY_VERSION$GBIF$citation` Accessed from <`r AMR:::TAXONOMY_VERSION$GBIF$url`> on `r AMR:::documentation_date(AMR:::TAXONOMY_VERSION$GBIF$accessed_date)`.
* `r AMR:::TAXONOMY_VERSION$SNOMED$citation` URL: <`r AMR:::TAXONOMY_VERSION$SNOMED$url`>

### Example content

Included (sub)species per taxonomic kingdom:

```{r, echo = FALSE}
microorganisms %>%
  count(kingdom) %>%
  mutate(n = format(n, big.mark = " ")) %>%
  setNames(c("Kingdom", "Number of (sub)species")) %>%
  print_df()
```

Example rows when filtering on genus *Escherichia*:

```{r, echo = FALSE}
microorganisms %>%
  filter(genus == "Escherichia") %>%
  print_df()
```


## `antibiotics`: Antibiotic (+Antifungal) Drugs

`r structure_txt(antibiotics)`

This data set is in R available as `antibiotics`, after you load the `AMR` package.

`r download_txt("antibiotics")`

The tab-separated text file and Microsoft Excel workbook, and SAS, SPSS and Stata files all contain the ATC codes, common abbreviations, trade names and LOINC codes as comma separated values.

### Source

This data set contains all EARS-Net and ATC codes gathered from WHO and WHONET, and all compound IDs from PubChem. It also contains all brand names (synonyms) as found on PubChem and Defined Daily Doses (DDDs) for oral and parenteral administration.

* [ATC/DDD index from WHO Collaborating Centre for Drug Statistics Methodology](https://atcddd.fhi.no/atc_ddd_index/) (note: this may not be used for commercial purposes, but is freely available from the WHO CC website for personal use)
* [PubChem by the US National Library of Medicine](https://pubchem.ncbi.nlm.nih.gov)
* [WHONET software 2019](https://whonet.org)
* [LOINC (Logical Observation Identifiers Names and Codes)](https://loinc.org)

### Example content

```{r, echo = FALSE}
antibiotics %>%
  filter(ab %in% colnames(example_isolates)) %>%
  print_df()
```


## `antivirals`: Antiviral Drugs

`r structure_txt(antivirals)`

This data set is in R available as `antivirals`, after you load the `AMR` package.

`r download_txt("antivirals")`

The tab-separated text file and Microsoft Excel workbook, and SAS, SPSS and Stata files all contain the trade names and LOINC codes as comma separated values.

### Source

This data set contains all ATC codes gathered from WHO and all compound IDs from PubChem. It also contains all brand names (synonyms) as found on PubChem and Defined Daily Doses (DDDs) for oral and parenteral administration.

* [ATC/DDD index from WHO Collaborating Centre for Drug Statistics Methodology](https://atcddd.fhi.no/atc_ddd_index/) (note: this may not be used for commercial purposes, but is freely available from the WHO CC website for personal use)
* [PubChem by the US National Library of Medicine](https://pubchem.ncbi.nlm.nih.gov)
* [LOINC (Logical Observation Identifiers Names and Codes)](https://loinc.org)

### Example content

```{r, echo = FALSE}
antivirals %>%
  print_df()
```

## `clinical_breakpoints`: Interpretation from MIC values & disk diameters to SIR

`r structure_txt(clinical_breakpoints)`

This data set is in R available as `clinical_breakpoints`, after you load the `AMR` package.

`r download_txt("clinical_breakpoints")`

### Source

This data set contains interpretation rules for MIC values and disk diffusion diameters. Included guidelines are CLSI (`r min(as.integer(gsub("[^0-9]", "", subset(clinical_breakpoints, guideline %like% "CLSI")$guideline)))`-`r max(as.integer(gsub("[^0-9]", "", subset(clinical_breakpoints, guideline %like% "CLSI")$guideline)))`) and EUCAST (`r min(as.integer(gsub("[^0-9]", "", subset(clinical_breakpoints, guideline %like% "EUCAST")$guideline)))`-`r max(as.integer(gsub("[^0-9]", "", subset(clinical_breakpoints, guideline %like% "EUCAST")$guideline)))`).

Clinical breakpoints in this package were validated through and imported from [WHONET](https://whonet.org), a free desktop Windows application developed and supported by the WHO Collaborating Centre for Surveillance of Antimicrobial Resistance. More can be read on [their website](https://whonet.org). The developers of WHONET and this `AMR` package have been in contact about sharing their work. We highly appreciate their development on the WHONET software.

The CEO of CLSI and the chairman of EUCAST have endorsed the work and public use of this `AMR` package (and consequently the use of their breakpoints) in June 2023, when future development of distributing clinical breakpoints was discussed in a meeting between CLSI, EUCAST, the WHO, and developers of WHONET and the `AMR` package.

**NOTE:** this `AMR` package (and the WHONET software as well) contains internal methods to apply the guidelines, which is rather complex. For example, some breakpoints must be applied on certain species groups (which are in case of this package available through the `microorganisms.groups` data set). It is important that this is considered when using the breakpoints for own use.

### Example content

```{r, echo = FALSE}
clinical_breakpoints %>%
  mutate(mo_name = mo_name(mo, language = NULL), .after = mo) %>%
  mutate(ab_name = ab_name(ab, language = NULL), .after = ab) %>%
  print_df()
```


## `intrinsic_resistant`: Intrinsic Bacterial Resistance

`r structure_txt(intrinsic_resistant)`

This data set is in R available as `intrinsic_resistant`, after you load the `AMR` package.

`r download_txt("intrinsic_resistant")`

### Source

This data set contains all defined intrinsic resistance by EUCAST of all bug-drug combinations, and is based on `r AMR:::format_eucast_version_nr("3.3")`.

### Example content

Example rows when filtering on *Enterobacter cloacae*:

```{r, echo = FALSE}
intrinsic_resistant %>%
  transmute(
    microorganism = mo_name(mo),
    antibiotic = ab_name(ab)
  ) %>%
  filter(microorganism == "Enterobacter cloacae") %>%
  arrange(antibiotic) %>%
  print_df(rows = Inf)
```


## `dosage`: Dosage Guidelines from EUCAST

`r structure_txt(dosage)`

This data set is in R available as `dosage`, after you load the `AMR` package.

`r download_txt("dosage")`

### Source

EUCAST breakpoints used in this package are based on the dosages in this data set.

Currently included dosages in the data set are meant for: `r AMR:::format_eucast_version_nr(unique(dosage$eucast_version))`.

### Example content

```{r, echo = FALSE}
dosage %>%
  print_df()
```


## `example_isolates`: Example Data for Practice

`r structure_txt(example_isolates)`

This data set is in R available as `example_isolates`, after you load the `AMR` package.

`r download_txt("example_isolates")`

### Source

This data set contains randomised fictitious data, but reflects reality and can be used to practise AMR data analysis.

### Example content

```{r, echo = FALSE}
example_isolates %>%
  print_df()
```

## `example_isolates_unclean`: Example Data for Practice

`r structure_txt(example_isolates_unclean)`

This data set is in R available as `example_isolates_unclean`, after you load the `AMR` package.

`r download_txt("example_isolates_unclean")`

### Source

This data set contains randomised fictitious data, but reflects reality and can be used to practise AMR data analysis.

### Example content

```{r, echo = FALSE}
example_isolates_unclean %>%
  print_df()
```


## `microorganisms.groups`: Species Groups and Microbiological Complexes

`r structure_txt(microorganisms.groups)`

This data set is in R available as `microorganisms.groups`, after you load the `AMR` package.

`r download_txt("microorganisms.groups")`

### Source

This data set contains species groups and microbiological complexes, which are used in the `clinical_breakpoints` data set.

### Example content

```{r, echo = FALSE}
microorganisms.groups %>%
  print_df()
```


## `microorganisms.codes`: Common Laboratory Codes

`r structure_txt(microorganisms.codes)`

This data set is in R available as `microorganisms.codes`, after you load the `AMR` package.

`r download_txt("microorganisms.codes")`

### Source

This data set contains commonly used codes for microorganisms, from laboratory systems and [WHONET](https://whonet.org).

### Example content

```{r, echo = FALSE}
microorganisms.codes %>%
  print_df()
```