(v0.7.1.9073) as.mo() self-learning algorithm

This commit is contained in:
dr. M.S. (Matthijs) Berends 2019-09-15 22:57:30 +02:00
parent cd178ee569
commit 398c5bdc4f
31 changed files with 1030 additions and 2360 deletions

View File

@ -21,3 +21,4 @@
^pkgdown$
^public$
^data-raw$
R/aa_test.R$

1
.gitignore vendored
View File

@ -23,3 +23,4 @@ packrat/src/
cran-comments.md
data-raw/taxon.tab
data-raw/DSMZ_bactnames.xlsx
R/aa_test.R

View File

@ -118,7 +118,7 @@ coverage:
# install missing and outdated packages
- Rscript -e 'source(".gitlab-ci.R"); gl_update_pkg_all(repos = "https://cran.rstudio.com", quiet = TRUE, install_pkgdown = FALSE)'
# codecov token is set in https://gitlab.com/msberends/AMR/settings/ci_cd
- Rscript -e "cc <- covr::package_coverage(line_exclusions = list('R/atc_online.R', 'R/mo_source.R')); covr::codecov(coverage = cc, token = '$codecov'); cat('Code coverage:', covr::percent_coverage(cc))"
- Rscript -e "cc <- covr::package_coverage(); covr::codecov(coverage = cc, token = '$codecov'); cat('Code coverage:', covr::percent_coverage(cc))"
coverage: '/Code coverage: \d+\.\d+/'
pages:

View File

@ -1,6 +1,6 @@
Package: AMR
Version: 0.7.1.9072
Date: 2019-09-12
Version: 0.7.1.9073
Date: 2019-09-15
Title: Antimicrobial Resistance Analysis
Authors@R: c(
person(role = c("aut", "cre"),
@ -17,6 +17,8 @@ Authors@R: c(
family = "Glasner", given = "Corinna", email = "c.glasner@umcg.nl", comment = c(ORCID = "0000-0003-1241-1328")),
person(role = "ctb",
family = "Hassing", given = c("Erwin", "E.", "A."), email = "e.hassing@certe.nl"),
person(role = "ctb",
family = "Hazenberg", given = c("Eric", "H.", "L.", "C.", "M."), email = "e.hazenberg@jbz.nl"),
person(role = "ctb",
family = "Lenglet", given = "Annick", email = "annick.lenglet@amsterdam.msf.org"),
person(role = "ctb",
@ -57,3 +59,4 @@ License: GPL-2 | file LICENSE
Encoding: UTF-8
LazyData: true
RoxygenNote: 6.1.1
StagedInstall: false

View File

@ -58,6 +58,7 @@ S3method(type_sum,mic)
S3method(type_sum,mo)
S3method(type_sum,rsi)
export("%like%")
export("%like_case%")
export(ab_atc)
export(ab_atc_group1)
export(ab_atc_group2)
@ -85,6 +86,7 @@ export(availability)
export(brmo)
export(bug_drug_combinations)
export(catalogue_of_life_version)
export(clear_mo_history)
export(count_I)
export(count_IR)
export(count_R)

19
NEWS.md
View File

@ -1,5 +1,5 @@
# AMR 0.7.1.9072
<small>Last updated: 12-Sep-2019</small>
# AMR 0.7.1.9073
<small>Last updated: 15-Sep-2019</small>
### Breaking
* Determination of first isolates now **excludes** all 'unknown' microorganisms at default, i.e. microbial code `"UNKNOWN"`. They can be included with the new parameter `include_unknown`:
@ -72,6 +72,14 @@
```
### Changed
* Many algorithm improvements for `as.mo()` (of which some led to additions to the `microorganisms` data set):
* Self-learning algorithm - the function now gains experience from previously determined microorganism IDs and learns from it (yielding 80-95% speed improvement for any guess after the first try)
* Big improvement for misspelled input
* These new trivial names known to the field are now understood: meningococcus, gonococcus, pneumococcus
* Updated to the latest taxonomic data (updated to August 2019, from the International Journal of Systematic and Evolutionary Microbiology
* Added support for Viridans Group Streptococci (VGS) and Milleri Group Streptococci (MGS)
* Added support for 5,000 new fungi
* Added support for unknown yeasts and fungi
* Renamed data set `septic_patients` to `example_isolates`
* Function `eucast_rules()`:
* Fixed a bug for *Yersinia pseudotuberculosis*
@ -83,13 +91,6 @@
* Removed class `atc` - using `as.atc()` is now deprecated in favour of `ab_atc()` and this will return a character, not the `atc` class anymore
* Removed deprecated functions `abname()`, `ab_official()`, `atc_name()`, `atc_official()`, `atc_property()`, `atc_tradenames()`, `atc_trivial_nl()`
* Fix and speed improvement for `mo_shortname()`
* Algorithm improvements for `as.mo()` (by which some additions were made to the `microorganisms` data set:
* Big improvement for misspelled input
* These new trivial names known to the field are now understood: meningococcus, gonococcus, pneumococcus
* Updated to the latest taxonomic data (updated to August 2019, from the International Journal of Systematic and Evolutionary Microbiology
* Added support for Viridans Group Streptococci (VGS) and Milleri Group Streptococci (MGS)
* Added support for 5,000 new fungi
* Added support for unknown yeasts and fungi
* Fix for using `mo_*` functions where the coercion uncertainties and failures would not be available through `mo_uncertainties()` and `mo_failures()` anymore
* Deprecated the `country` parameter of `mdro()` in favour of the already existing `guideline` parameter to support multiple guidelines within one country
* The `name` of `RIF` is now Rifampicin instead of Rifampin

View File

@ -21,7 +21,7 @@
#' Pattern Matching
#'
#' Convenient wrapper around \code{\link[base]{grep}} to match a pattern: \code{a \%like\% b}. It always returns a \code{logical} vector and is always case-insensitive. Also, \code{pattern} (\code{b}) can be as long as \code{x} (\code{a}) to compare items of each index in both vectors.
#' Convenient wrapper around \code{\link[base]{grep}} to match a pattern: \code{a \%like\% b}. It always returns a \code{logical} vector and is always case-insensitive (use \code{a \%like_case\% b} for case-sensitive matching). Also, \code{pattern} (\code{b}) can be as long as \code{x} (\code{a}) to compare items of each index in both vectors, or can both have the same length to iterate over all cases.
#' @inheritParams base::grepl
#' @return A \code{logical} vector
#' @name like
@ -53,14 +53,14 @@
#' left_join_microorganisms() %>%
#' filter(genus %like% '^ent') %>%
#' freq(genus, species)
like <- function(x, pattern) {
like <- function(x, pattern, ignore.case = TRUE) {
if (length(pattern) > 1) {
if (length(x) != length(pattern)) {
if (length(x) == 1) {
x <- rep(x, length(pattern))
}
# return TRUE for every 'x' that matches any 'pattern', FALSE otherwise
res <- sapply(pattern, function(pttrn) x %like% pttrn)
res <- sapply(pattern, function(pttrn) base::grepl(pttrn, x, ignore.case = ignore.case))
res2 <- as.logical(rowSums(res))
# get only first item of every hit in pattern
res2[duplicated(res)] <- FALSE
@ -71,9 +71,9 @@ like <- function(x, pattern) {
res <- vector(length = length(pattern))
for (i in 1:length(res)) {
if (is.factor(x[i])) {
res[i] <- as.integer(x[i]) %in% base::grep(pattern[i], levels(x[i]), ignore.case = TRUE)
res[i] <- as.integer(x[i]) %in% base::grep(pattern[i], levels(x[i]), ignore.case = ignore.case)
} else {
res[i] <- base::grepl(pattern[i], x[i], ignore.case = TRUE)
res[i] <- base::grepl(pattern[i], x[i], ignore.case = ignore.case)
}
}
return(res)
@ -82,16 +82,24 @@ like <- function(x, pattern) {
# the regular way how grepl works; just one pattern against one or more x
if (is.factor(x)) {
as.integer(x) %in% base::grep(pattern, levels(x), ignore.case = TRUE)
as.integer(x) %in% base::grep(pattern, levels(x), ignore.case = ignore.case)
} else {
tryCatch(base::grepl(pattern, x, ignore.case = TRUE),
tryCatch(base::grepl(pattern, x, ignore.case = ignore.case),
error = function(e) ifelse(test = grepl("Invalid regexp", e$message),
# try with perl = TRUE:
yes = return(base::grepl(pattern, x, ignore.case = TRUE, perl = TRUE)),
yes = return(base::grepl(pattern, x, ignore.case = ignore.case, perl = TRUE)),
no = stop(e$message)))
}
}
#' @rdname like
#' @export
"%like%" <- like
"%like%" <- function(x, pattern) {
like(x, pattern, ignore.case = TRUE)
}
#' @rdname like
#' @export
"%like_case%" <- function(x, pattern) {
like(x, pattern, ignore.case = FALSE)
}

View File

@ -41,8 +41,8 @@ percent <- function(x, round = 1, force_zero = FALSE, decimal.mark = getOption("
big.mark <- " "
}
}
x <- percent_clean(x = x, round = round, force_zero = force_zero,
decimal.mark = decimal.mark, big.mark = big.mark, ...)
percent_clean(x = x, round = round, force_zero = force_zero,
decimal.mark = decimal.mark, big.mark = big.mark, ...)
}
#' @importFrom crayon blue bold red

1431
R/mo.R

File diff suppressed because it is too large Load Diff

1489
R/mo2.R

File diff suppressed because it is too large Load Diff

View File

@ -19,116 +19,152 @@
# Visit our website for more info: https://msberends.gitlab.io/AMR. #
# ==================================================================== #
# print successful as.mo coercions to AMR environment
#' @importFrom dplyr %>% distinct filter
set_mo_history <- function(x, mo, uncertainty_level, force = FALSE) {
# disable function for now
return(base::invisible())
mo_history_file <- file.path(file.path(system.file(package = "AMR"), "mo_history"), "mo_history.csv")
# if (base::interactive() | force == TRUE) {
# mo_hist <- read_mo_history(uncertainty_level = uncertainty_level, force = force)
# df <- data.frame(x, mo, stringsAsFactors = FALSE) %>%
# distinct(x, .keep_all = TRUE) %>%
# filter(!is.na(x) & !is.na(mo))
# if (nrow(df) == 0) {
# return(base::invisible())
# }
# x <- toupper(df$x)
# mo <- df$mo
# for (i in 1:length(x)) {
# # save package version too, as both the as.mo() algorithm and the reference data set may change
# if (NROW(mo_hist[base::which(mo_hist$x == x[i] &
# mo_hist$uncertainty_level >= uncertainty_level &
# mo_hist$package_v == utils::packageVersion("AMR")),]) == 0) {
# tryCatch(
# assign(x = "mo_history",
# value = rbind(mo_hist,
# data.frame(
# x = x[i],
# mo = mo[i],
# uncertainty_level = uncertainty_level,
# package_v = base::as.character(utils::packageVersion("AMR")),
# stringsAsFactors = FALSE)),
# envir = asNamespace("AMR")),
# error = function(e) invisible())
# }
# }
# }
# return(base::invisible())
# print successful as.mo coercions to a options entry
#' @importFrom dplyr %>% distinct filter
set_mo_history <- function(x, mo, uncertainty_level, force = FALSE, disable = FALSE) {
if (isTRUE(disable)) {
return(base::invisible())
}
if (base::interactive() | force == TRUE) {
mo_hist <- read_mo_history(uncertainty_level = uncertainty_level, force = force)
df <- data.frame(x, mo, stringsAsFactors = FALSE) %>%
distinct(x, .keep_all = TRUE) %>%
filter(!is.na(x) & !is.na(mo))
if (nrow(df) == 0) {
return(base::invisible())
}
x <- toupper(df$x)
mo <- df$mo
for (i in 1:length(x)) {
# save package version too, as both the as.mo() algorithm and the reference data set may change
if (NROW(mo_hist[base::which(mo_hist$x == x[i] &
mo_hist$uncertainty_level >= uncertainty_level &
mo_hist$package_v == utils::packageVersion("AMR")),]) == 0) {
# # Not using the file system:
# tryCatch(options(mo_remembered_results = rbind(mo_hist,
# data.frame(
# x = x[i],
# mo = mo[i],
# uncertainty_level = uncertainty_level,
# package_v = base::as.character(utils::packageVersion("AMR")),
# stringsAsFactors = FALSE))),
# error = function(e) base::invisible())
# # don't remember more than 1,000 different input values
# if (tryCatch(nrow(getOption("mo_remembered_results")), error = function(e) 1001) > 1000) {
# return(base::invisible())
# }
if (is.null(mo_hist)) {
message(blue(paste0("NOTE: results are saved to ", mo_history_file, ".")))
}
tryCatch(write.csv(rbind(mo_hist,
data.frame(
x = x[i],
mo = mo[i],
uncertainty_level = uncertainty_level,
package_v = base::as.character(utils::packageVersion("AMR")),
stringsAsFactors = FALSE)),
file = mo_history_file, row.names = FALSE),
error = function(e) base::invisible())
}
}
}
return(base::invisible())
}
get_mo_history <- function(x, uncertainty_level, force = FALSE) {
# disable function for now
return(NA)
get_mo_history <- function(x, uncertainty_level, force = FALSE, disable = FALSE) {
if (isTRUE(disable)) {
return(to_class_mo(NA))
}
# history <- read_mo_history(uncertainty_level = uncertainty_level, force = force)
# if (base::is.null(history)) {
# NA
# } else {
# data.frame(x = toupper(x), stringsAsFactors = FALSE) %>%
# left_join(history, by = "x") %>%
# pull(mo)
# }
history <- read_mo_history(uncertainty_level = uncertainty_level, force = force)
if (base::is.null(history)) {
result <- NA
} else {
result <- data.frame(x = toupper(x), stringsAsFactors = FALSE) %>%
left_join(history, by = "x") %>%
pull(mo)
}
to_class_mo(result)
}
#' @importFrom dplyr %>% filter distinct
read_mo_history <- function(uncertainty_level = 2, force = FALSE, unfiltered = FALSE) {
# disable function for now
return(NULL)
read_mo_history <- function(uncertainty_level = 2, force = FALSE, unfiltered = FALSE, disable = FALSE) {
if (isTRUE(disable)) {
return(NULL)
}
# if ((!base::interactive() & force == FALSE)) {
# return(NULL)
# }
# uncertainty_level_param <- uncertainty_level
#
# history <- tryCatch(get("mo_history", envir = asNamespace("AMR")),
# error = function(e) NULL)
# if (is.null(history)) {
# return(NULL)
# }
# # Below: filter on current package version.
# # Even current fullnames may be replaced by new taxonomic names, so new versions of
# # the Catalogue of Life must not lead to data corruption.
#
# if (unfiltered == FALSE) {
# history <- history %>%
# filter(package_v == as.character(utils::packageVersion("AMR")),
# # only take unknowns if uncertainty_level_param is higher
# ((mo == "UNKNOWN" & uncertainty_level_param == uncertainty_level) |
# (mo != "UNKNOWN" & uncertainty_level_param >= uncertainty_level))) %>%
# arrange(desc(uncertainty_level)) %>%
# distinct(x, mo, .keep_all = TRUE)
# }
#
# if (nrow(history) == 0) {
# NULL
# } else {
# history
# }
if ((!base::interactive() & force == FALSE)) {
return(NULL)
}
uncertainty_level_param <- uncertainty_level
# # Not using the file system:
# history <- tryCatch(getOption("mo_remembered_results"),
# error = function(e) NULL)
history <- tryCatch(read.csv(mo_history_file, stringsAsFactors = FALSE),
warning = function(w) invisible(),
error = function(e) NULL)
if (is.null(history)) {
return(NULL)
}
# Below: filter on current package version.
# Even current fullnames may be replaced by new taxonomic names, so new versions of
# the Catalogue of Life must not lead to data corruption.
if (unfiltered == FALSE) {
history <- history %>%
filter(package_v == as.character(utils::packageVersion("AMR")),
# only take unknowns if uncertainty_level_param is higher
((mo == "UNKNOWN" & uncertainty_level_param == uncertainty_level) |
(mo != "UNKNOWN" & uncertainty_level_param >= uncertainty_level))) %>%
arrange(desc(uncertainty_level)) %>%
distinct(x, mo, .keep_all = TRUE)
}
if (nrow(history) == 0) {
NULL
} else {
history
}
}
# @rdname as.mo
# @importFrom crayon red
# @importFrom utils menu
# @export
clean_mo_history <- function(...) {
# if (!is.null(read_mo_history())) {
# if (interactive() & !isTRUE(list(...)$force)) {
# q <- menu(title = paste("This will remove all",
# format(nrow(read_mo_history(999, unfiltered = TRUE)), big.mark = ","),
# "microbial IDs determined previously in this session. Are you sure?"),
# choices = c("Yes", "No"),
# graphics = FALSE)
# if (q != 1) {
# return(invisible())
# }
# }
# tryCatch(
# assign(x = "mo_history",
# value = NULL,
# envir = asNamespace("AMR")),
# error = function(e) invisible())
# cat(red("History removed."))
# }
#' @rdname as.mo
#' @importFrom crayon red
#' @importFrom utils menu
#' @export
clear_mo_history <- function(...) {
if (!is.null(read_mo_history())) {
if (interactive() & !isTRUE(list(...)$force)) {
q <- menu(title = paste("This will clear all",
format(nrow(read_mo_history(999, unfiltered = TRUE)), big.mark = ","),
"previously determined microbial IDs. Are you sure?"),
choices = c("Yes", "No"),
graphics = FALSE)
if (q != 1) {
return(invisible())
}
}
# # Not using the file system:
# success <- tryCatch(options(mo_remembered_results = NULL),
# error = function(e) FALSE)
success <- create_blank_mo_history()
if (!isFALSE(success)) {
cat(red(paste("File", mo_history_file, "cleared.")))
}
}
}
create_blank_mo_history <- function() {
tryCatch(
write.csv(x = data.frame(x = character(0),
mo = character(0),
uncertainty_level = integer(0),
package_v = character(0),
stringsAsFactors = FALSE),
file = mo_history_file),
warning = function(w) invisible(),
error = function(e) TRUE)
}

View File

@ -408,7 +408,7 @@ mo_validate <- function(x, property, ...) {
if (!"AMR" %in% base::.packages()) {
require("AMR")
# check onLoad() in R/zzz.R: data tables are created there.
# check .onLoad() in R/zzz.R: data tables are created there.
}
# try to catch an error when inputting an invalid parameter

View File

@ -26,7 +26,8 @@
# register data
microorganisms.oldDT <- as.data.table(AMR::microorganisms.old)
microorganisms.oldDT$fullname_lower <- tolower(microorganisms.oldDT$fullname)
# for fullname_lower: keep only dots, letters, numbers, slashes, spaces and dashes
microorganisms.oldDT$fullname_lower <- gsub("[^.a-z0-9/ \\-]+", "", tolower(microorganisms.oldDT$fullname))
setkey(microorganisms.oldDT, col_id, fullname)
assign(x = "microorganismsDT",
@ -81,7 +82,8 @@
#' @importFrom data.table as.data.table setkey
make_DT <- function() {
microorganismsDT <- as.data.table(AMR::microorganisms)
microorganismsDT$fullname_lower <- tolower(microorganismsDT$fullname)
# for fullname_lower: keep only dots, letters, numbers, slashes, spaces and dashes
microorganismsDT$fullname_lower <- gsub("[^.a-z0-9/ \\-]+", "", tolower(microorganismsDT$fullname))
setkey(microorganismsDT,
prevalence,
kingdom,

View File

@ -118,7 +118,7 @@ reference:
- "`ab_property`"
- "`age`"
- "`age_groups`"
- "`p.symbol`"
- "`p_symbol`"
- "`join`"
- "`atc_online_property`"
- title: "Analysing your data"

View File

@ -19,3 +19,4 @@ ignore:
- "R/atc_online.R"
- "R/mo_source.R"
- "R/resistance_predict.R"
- "R/aa_test.R"

View File

@ -19,7 +19,7 @@ eucast_rules_file <- dplyr::arrange(
reference.rule_group,
reference.rule)
# Translations -----
# Translations ----
translations_file <- utils::read.delim(file = "data-raw/translations.tsv",
sep = "\t",
stringsAsFactors = FALSE,
@ -42,3 +42,16 @@ usethis::use_data(eucast_rules_file, translations_file,
# Remove from global environment ----
rm(eucast_rules_file)
rm(translations_file)
# Clean mo history ----
mo_history_file <- file.path(file.path(system.file(package = "AMR"), "mo_history"), "mo_history.csv")
usethis::ui_done(paste0("Resetting {usethis::ui_value('", mo_history_file, "')}"))
tryCatch(
write.csv(x = data.frame(x = character(0),
mo = character(0),
uncertainty_level = integer(0),
package_v = character(0),
stringsAsFactors = FALSE),
file = mo_history_file),
warning = function(w) invisible(),
error = function(e) TRUE)

View File

@ -78,7 +78,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>

Binary file not shown.

Before

Width:  |  Height:  |  Size: 219 KiB

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 103 KiB

After

Width:  |  Height:  |  Size: 60 KiB

View File

@ -78,7 +78,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>

View File

@ -78,7 +78,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>
@ -253,6 +253,10 @@
<p><strong>Erwin E. A. Hassing</strong>. Contributor.
</p>
</li>
<li>
<p><strong>Eric H. L. C. M. Hazenberg</strong>. Contributor.
</p>
</li>
<li>
<p><strong>Annick Lenglet</strong>. Contributor.
</p>

View File

@ -42,7 +42,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>

View File

@ -80,7 +80,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9055</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>
@ -243,7 +243,9 @@
<span class='fu'>mo_uncertainties</span>()
<span class='fu'>mo_renamed</span>()</pre>
<span class='fu'>mo_renamed</span>()
<span class='fu'>clear_mo_history</span>(<span class='no'>...</span>)</pre>
<h2 class="hasAnchor" id="arguments"><a class="anchor" href="#arguments"></a>Arguments</h2>
<table class="ref-arguments">
@ -283,7 +285,7 @@
<h2 class="hasAnchor" id="details"><a class="anchor" href="#details"></a>Details</h2>
<p><strong>General info</strong> <br />
A microbial ID from this package (class: <code>mo</code>) typically looks like these examples:<br /></p><pre>
A microorganism ID from this package (class: <code>mo</code>) typically looks like these examples:<br /></p><pre>
Code Full name
--------------- --------------------------------------
B_KLBSL Klebsiella
@ -299,8 +301,11 @@ A microbial ID from this package (class: <code>mo</code>) typically looks like t
</pre>
<p>Values that cannot be coered will be considered 'unknown' and will get the MO code <code>UNKNOWN</code>.</p>
<p>Use the <code><a href='mo_property.html'>mo_property</a>_*</code> functions to get properties based on the returned code, see Examples.</p>
<p>The algorithm uses data from the Catalogue of Life (see below) and from one other source (see <code><a href='microorganisms.html'>?microorganisms</a></code>).</p>
<p><strong>Intelligent rules</strong> <br />
<p>The algorithm uses data from the Catalogue of Life (see below) and from one other source (see <code><a href='microorganisms.html'>microorganisms</a></code>).</p>
<p><strong>Self-learning algoritm</strong> <br />
The <code>as.mo()</code> function gains experience from previously determined microorganism IDs and learns from it. This drastically improves both speed and reliability. Use <code>clear_mo_history()</code> to reset the algorithms. Only experience from your current <code>AMR</code> package version is used. This is done because in the future the taxonomic tree (which is included in this package) may change for any organism and it consequently has to rebuild its knowledge.</p>
<p>Usually, any guess after the first try runs 80-95% faster than the first try.</p>
<p><strong>Intelligent rules</strong> <br />
This function uses intelligent rules to help getting fast and logical results. It tries to find matches in this order:</p><ul>
<li><p>Valid MO codes and full names: it first searches in already valid MO code and known genus/species combinations</p></li>
<li><p>Human pathogenic prevalence: it first searches in more prevalent microorganisms, then less prevalent ones (see <em>Microbial prevalence of pathogens in humans</em> below)</p></li>
@ -326,7 +331,7 @@ The algorithm can additionally use three different levels of uncertainty to gues
</ul>
<p>Use <code>mo_failures()</code> to get a vector with all values that could not be coerced to a valid value.</p>
<p>Use <code>mo_uncertainties()</code> to get a data.frame with all values that were coerced to a valid value, but with uncertainty.</p>
<p>Use <code>mo_renamed()</code> to get a vector with all values that could be coerced based on an old, previously accepted taxonomic name.</p>
<p>Use <code>mo_renamed()</code> to get a data.frame with all values that could be coerced based on an old, previously accepted taxonomic name.</p>
<p><strong>Microbial prevalence of pathogens in humans</strong> <br />
The intelligent rules take into account microbial prevalence of pathogens in humans. It uses three groups and all (sub)species are in only one group. These groups are:</p><ul>
<li><p>1 (most prevalent): class is Gammaproteobacteria <strong>or</strong> genus is one of: <em>Enterococcus</em>, <em>Staphylococcus</em>, <em>Streptococcus</em>.</p></li>
@ -334,7 +339,7 @@ The intelligent rules take into account microbial prevalence of pathogens in hum
<li><p>3 (least prevalent): all others.</p></li>
</ul>
<p>Group 1 contains all common Gram positives and Gram negatives, like all Enterobacteriaceae and e.g. <em>Pseudomonas</em> and <em>Legionella</em>.</p>
<p>Group 2 probably contains less microbial pathogens; all other members of phyla that were found in humans in the Northern Netherlands between 2001 and 2018.</p>
<p>Group 2 contains probably less pathogenic microorganisms; all other members of phyla that were found in humans in the Northern Netherlands between 2001 and 2018.</p>
<h2 class="hasAnchor" id="source"><a class="anchor" href="#source"></a>Source</h2>

View File

@ -78,7 +78,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>
@ -292,7 +292,7 @@
</tr><tr>
<td>
<p><code><a href="as.mo.html">as.mo()</a></code> <code><a href="as.mo.html">is.mo()</a></code> <code><a href="as.mo.html">mo_failures()</a></code> <code><a href="as.mo.html">mo_uncertainties()</a></code> <code><a href="as.mo.html">mo_renamed()</a></code> </p>
<p><code><a href="as.mo.html">as.mo()</a></code> <code><a href="as.mo.html">is.mo()</a></code> <code><a href="as.mo.html">mo_failures()</a></code> <code><a href="as.mo.html">mo_uncertainties()</a></code> <code><a href="as.mo.html">mo_renamed()</a></code> <code><a href="as.mo.html">clear_mo_history()</a></code> </p>
</td>
<td><p>Transform to microorganism ID</p></td>
</tr><tr>
@ -384,9 +384,9 @@
</tr><tr>
<td>
<p><code><a href="AMR-deprecated.html">as.atc()</a></code> <code><a href="AMR-deprecated.html">p.symbol()</a></code> </p>
<p><code><a href="p_symbol.html">p_symbol()</a></code> </p>
</td>
<td><p>Deprecated functions</p></td>
<td><p>Symbol of a p value</p></td>
</tr><tr>
<td>
@ -528,7 +528,7 @@
</tr><tr>
<td>
<p><code><a href="like.html">like()</a></code> <code><a href="like.html">`%like%`</a></code> </p>
<p><code><a href="like.html">like()</a></code> <code><a href="like.html">`%like%`</a></code> <code><a href="like.html">`%like_case%`</a></code> </p>
</td>
<td><p>Pattern Matching</p></td>
</tr>

View File

@ -47,7 +47,7 @@
<script src="../extra.js"></script>
<meta property="og:title" content="Pattern Matching — like" />
<meta property="og:description" content="Convenient wrapper around grep to match a pattern: a %like% b. It always returns a logical vector and is always case-insensitive. Also, pattern (b) can be as long as x (a) to compare items of each index in both vectors." />
<meta property="og:description" content="Convenient wrapper around grep to match a pattern: a %like% b. It always returns a logical vector and is always case-insensitive (use a %like_case% b for case-sensitive matching). Also, pattern (b) can be as long as x (a) to compare items of each index in both vectors, or can both have the same length to iterate over all cases." />
<meta property="og:image" content="https://msberends.gitlab.io/AMR/logo.png" />
<meta name="twitter:card" content="summary" />
@ -80,7 +80,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9067</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>
@ -230,13 +230,15 @@
<div class="ref-description">
<p>Convenient wrapper around <code><a href='https://www.rdocumentation.org/packages/base/topics/grep'>grep</a></code> to match a pattern: <code>a %like% b</code>. It always returns a <code>logical</code> vector and is always case-insensitive. Also, <code>pattern</code> (<code>b</code>) can be as long as <code>x</code> (<code>a</code>) to compare items of each index in both vectors.</p>
<p>Convenient wrapper around <code><a href='https://www.rdocumentation.org/packages/base/topics/grep'>grep</a></code> to match a pattern: <code>a %like% b</code>. It always returns a <code>logical</code> vector and is always case-insensitive (use <code>a %like_case% b</code> for case-sensitive matching). Also, <code>pattern</code> (<code>b</code>) can be as long as <code>x</code> (<code>a</code>) to compare items of each index in both vectors, or can both have the same length to iterate over all cases.</p>
</div>
<pre class="usage"><span class='fu'>like</span>(<span class='no'>x</span>, <span class='no'>pattern</span>)
<pre class="usage"><span class='fu'>like</span>(<span class='no'>x</span>, <span class='no'>pattern</span>, <span class='kw'>ignore.case</span> <span class='kw'>=</span> <span class='fl'>TRUE</span>)
<span class='no'>x</span> <span class='kw'>%like%</span> <span class='no'>pattern</span></pre>
<span class='no'>x</span> <span class='kw'>%like%</span> <span class='no'>pattern</span>
<span class='no'>x</span> <span class='kw'>%like_case%</span> <span class='no'>pattern</span></pre>
<h2 class="hasAnchor" id="arguments"><a class="anchor" href="#arguments"></a>Arguments</h2>
<table class="ref-arguments">
@ -257,6 +259,11 @@
is used with a warning. Missing values are allowed except for
<code>regexpr</code> and <code>gregexpr</code>.</p></td>
</tr>
<tr>
<th>ignore.case</th>
<td><p>if <code>FALSE</code>, the pattern matching is <em>case
sensitive</em> and if <code>TRUE</code>, case is ignored during matching.</p></td>
</tr>
</table>
<h2 class="hasAnchor" id="source"><a class="anchor" href="#source"></a>Source</h2>

View File

@ -80,7 +80,7 @@
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">AMR (for R)</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9072</span>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Latest development version">0.7.1.9073</span>
</span>
</div>

View File

@ -0,0 +1 @@
"","x","mo","uncertainty_level","package_v"
1 x mo uncertainty_level package_v

View File

@ -1,5 +1,5 @@
% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/mo.R
% Please edit documentation in R/mo.R, R/mo_history.R
\name{as.mo}
\alias{as.mo}
\alias{mo}
@ -7,6 +7,7 @@
\alias{mo_failures}
\alias{mo_uncertainties}
\alias{mo_renamed}
\alias{clear_mo_history}
\title{Transform to microorganism ID}
\usage{
as.mo(x, Becker = FALSE, Lancefield = FALSE, allow_uncertain = TRUE,
@ -19,6 +20,8 @@ mo_failures()
mo_uncertainties()
mo_renamed()
clear_mo_history(...)
}
\arguments{
\item{x}{a character vector or a \code{data.frame} with one or two columns}
@ -45,7 +48,7 @@ Use this function to determine a valid microorganism ID (\code{mo}). Determinati
}
\details{
\strong{General info} \cr
A microbial ID from this package (class: \code{mo}) typically looks like these examples:\cr
A microorganism ID from this package (class: \code{mo}) typically looks like these examples:\cr
\preformatted{
Code Full name
--------------- --------------------------------------
@ -65,7 +68,13 @@ Values that cannot be coered will be considered 'unknown' and will get the MO co
Use the \code{\link{mo_property}_*} functions to get properties based on the returned code, see Examples.
The algorithm uses data from the Catalogue of Life (see below) and from one other source (see \code{?microorganisms}).
The algorithm uses data from the Catalogue of Life (see below) and from one other source (see \code{\link{microorganisms}}).
\strong{Self-learning algoritm} \cr
The \code{as.mo()} function gains experience from previously determined microorganism IDs and learns from it. This drastically improves both speed and reliability. Use \code{clear_mo_history()} to reset the algorithms. Only experience from your current \code{AMR} package version is used. This is done because in the future the taxonomic tree (which is included in this package) may change for any organism and it consequently has to rebuild its knowledge.
Usually, any guess after the first try runs 80-95\% faster than the first try.
\strong{Intelligent rules} \cr
This function uses intelligent rules to help getting fast and logical results. It tries to find matches in this order:
@ -105,7 +114,7 @@ Use \code{mo_failures()} to get a vector with all values that could not be coerc
Use \code{mo_uncertainties()} to get a data.frame with all values that were coerced to a valid value, but with uncertainty.
Use \code{mo_renamed()} to get a vector with all values that could be coerced based on an old, previously accepted taxonomic name.
Use \code{mo_renamed()} to get a data.frame with all values that could be coerced based on an old, previously accepted taxonomic name.
\strong{Microbial prevalence of pathogens in humans} \cr
The intelligent rules take into account microbial prevalence of pathogens in humans. It uses three groups and all (sub)species are in only one group. These groups are:
@ -117,7 +126,7 @@ The intelligent rules take into account microbial prevalence of pathogens in hum
Group 1 contains all common Gram positives and Gram negatives, like all Enterobacteriaceae and e.g. \emph{Pseudomonas} and \emph{Legionella}.
Group 2 probably contains less microbial pathogens; all other members of phyla that were found in humans in the Northern Netherlands between 2001 and 2018.
Group 2 contains probably less pathogenic microorganisms; all other members of phyla that were found in humans in the Northern Netherlands between 2001 and 2018.
}
\section{Source}{

View File

@ -3,14 +3,17 @@
\name{like}
\alias{like}
\alias{\%like\%}
\alias{\%like_case\%}
\title{Pattern Matching}
\source{
Idea from the \href{https://github.com/Rdatatable/data.table/blob/master/R/like.R}{\code{like} function from the \code{data.table} package}, but made it case insensitive at default and let it support multiple patterns. Also, if the regex fails the first time, it tries again with \code{perl = TRUE}.
}
\usage{
like(x, pattern)
like(x, pattern, ignore.case = TRUE)
x \%like\% pattern
x \%like_case\% pattern
}
\arguments{
\item{x}{a character vector where matches are sought, or an
@ -24,12 +27,15 @@ x \%like\% pattern
character vector of length 2 or more is supplied, the first element
is used with a warning. Missing values are allowed except for
\code{regexpr} and \code{gregexpr}.}
\item{ignore.case}{if \code{FALSE}, the pattern matching is \emph{case
sensitive} and if \code{TRUE}, case is ignored during matching.}
}
\value{
A \code{logical} vector
}
\description{
Convenient wrapper around \code{\link[base]{grep}} to match a pattern: \code{a \%like\% b}. It always returns a \code{logical} vector and is always case-insensitive. Also, \code{pattern} (\code{b}) can be as long as \code{x} (\code{a}) to compare items of each index in both vectors.
Convenient wrapper around \code{\link[base]{grep}} to match a pattern: \code{a \%like\% b}. It always returns a \code{logical} vector and is always case-insensitive (use \code{a \%like_case\% b} for case-sensitive matching). Also, \code{pattern} (\code{b}) can be as long as \code{x} (\code{a}) to compare items of each index in both vectors, or can both have the same length to iterate over all cases.
}
\details{
Using RStudio? This function can also be inserted from the Addins menu and can have its own Keyboard Shortcut like Ctrl+Shift+L or Cmd+Shift+L (see Tools > Modify Keyboard Shortcuts...).

View File

@ -22,6 +22,8 @@
context("mo.R")
test_that("as.mo works", {
clear_mo_history(force = TRUE)
library(dplyr)
MOs <- AMR::microorganisms %>% filter(!is.na(mo), nchar(mo) > 3)

View File

@ -20,7 +20,7 @@ knitr::opts_chunk$set(
comment = "#",
fig.width = 7.5,
fig.height = 4.5,
dpi = 150
dpi = 75
)
```
@ -110,26 +110,40 @@ That takes `r round(mean(T.islandicus$time, na.rm = TRUE) / mean(S.aureus$time,
In the figure below, we compare *Escherichia coli* (which is very common) with *Prevotella brevis* (which is moderately common) and with *Thermus islandicus* (which is uncommon):
```{r, echo = FALSE}
ggplot.bm(
microbenchmark(as.mo("Escherichia coli"),
as.mo("E. coli"),
times = 10), title = "Very common")
# ggplot.bm(
# microbenchmark(as.mo("Escherichia coli"),
# as.mo("E. coli"),
# times = 10), title = "Very common")
#
# ggplot.bm(
# microbenchmark(as.mo("Prevotella brevis"),
# as.mo("P. brevis"),
# times = 10), title = "Moderately common")
#
# ggplot.bm(
# microbenchmark(as.mo("Thermus islandicus"),
# as.mo("T. islandicus"),
# times = 10), title = "Uncommon")
ggplot.bm(
microbenchmark(as.mo("Prevotella brevis"),
as.mo("P. brevis"),
times = 10), title = "Moderately common")
ggplot.bm(
microbenchmark(as.mo("Thermus islandicus"),
as.mo("T. islandicus"),
times = 10), title = "Uncommon")
par(mar = c(5, 16, 4, 2))
boxplot(microbenchmark(
'as.mo("Thermus islandicus")' = as.mo("Thermus islandicus"),
'as.mo("Prevotella brevis")' = as.mo("Prevotella brevis"),
'as.mo("Escherichia coli")' = as.mo("Escherichia coli"),
'as.mo("T. islandicus")' = as.mo("T. islandicus"),
'as.mo("P. brevis")' = as.mo("P. brevis"),
'as.mo("E. coli")' = as.mo("E. coli"),
times = 10),
horizontal = TRUE, las = 1, unit = "s", log = FALSE,
xlab = "", ylab = "Time in seconds", ylim = c(0, 0.5),
main = "Benchmarks per prevalence")
```
```{r, echo = FALSE, eval = FALSE}
# In reality, the `as.mo()` functions **learns from its own output to speed up determinations for next times**. In above figure, this effect was disabled to show the difference with the boxplot below - when you would use `as.mo()` yourself:
In reality, the `as.mo()` functions **learns from its own output to speed up determinations for next times**. In above figure, this effect was disabled to show the difference with the boxplot below - when you would use `as.mo()` yourself:
clean_mo_history()
```{r, echo = FALSE}
clear_mo_history()
par(mar = c(5, 16, 4, 2))
boxplot(microbenchmark(
'as.mo("Thermus islandicus")' = as.mo("Thermus islandicus", force_mo_history = TRUE),
@ -142,10 +156,10 @@ boxplot(microbenchmark(
horizontal = TRUE, las = 1, unit = "s", log = FALSE,
xlab = "", ylab = "Time in seconds", ylim = c(0, 0.5),
main = "Benchmarks per prevalence")
# The highest outliers are the first times. All next determinations were done in only thousands of seconds. For now, learning only works per session. If R is closed or terminated, the algorithms reset. This will probably be resolved in a next version.
```
The highest outliers are the first times. All next determinations were done in only thousands of seconds.
Uncommon microorganisms take a lot more time than common microorganisms. To relieve this pitfall and further improve performance, two important calculations take almost no time at all: **repetitive results** and **already precalculated results**.
### Repetitive results