From 082e52a0dd5944911d5dd382758623c09ac251fb Mon Sep 17 00:00:00 2001 From: "Matthijs S. Berends" Date: Tue, 4 Oct 2022 11:20:58 +0200 Subject: [PATCH] new mo_cleaning_regex --- DESCRIPTION | 4 +- NAMESPACE | 1 + NEWS.md | 2 +- R/mo.R | 122 ++++++++++++++------------------------- R/mo_matching_score.R | 21 +++---- R/mo_property.R | 4 +- inst/tinytest/test-mo.R | 13 ----- man/as.mo.Rd | 54 +++++------------ man/mo_matching_score.Rd | 5 +- man/mo_property.Rd | 2 +- 10 files changed, 80 insertions(+), 148 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index 05852722..beeeb61b 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: AMR -Version: 1.8.2.9025 -Date: 2022-10-03 +Version: 1.8.2.9026 +Date: 2022-10-04 Title: Antimicrobial Resistance Data Analysis Description: Functions to simplify and standardise antimicrobial resistance (AMR) data analysis and to work with microbial and antimicrobial properties by diff --git a/NAMESPACE b/NAMESPACE index c652e0ae..03a2dbe0 100755 --- a/NAMESPACE +++ b/NAMESPACE @@ -266,6 +266,7 @@ export(mdro) export(mean_amr_distance) export(mo_authors) export(mo_class) +export(mo_cleaning_regex) export(mo_domain) export(mo_failures) export(mo_family) diff --git a/NEWS.md b/NEWS.md index f86578fd..6f4994e5 100755 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -# AMR 1.8.2.9025 +# AMR 1.8.2.9026 This version will eventually become v2.0! We're happy to reach a new major milestone soon! diff --git a/R/mo.R b/R/mo.R index 734fdff0..f85d948a 100755 --- a/R/mo.R +++ b/R/mo.R @@ -38,19 +38,16 @@ #' #' This excludes enterococci at default (who are in group D), use `Lancefield = "all"` to also categorise all enterococci as group D. #' @param minimum_matching_score a numeric value to set as the lower limit for the [MO matching score][mo_matching_score()]. When left blank, this will be determined automatically based on the character length of `x`, its [taxonomic kingdom][microorganisms] and [human pathogenicity][mo_matching_score()]. -#' @param allow_uncertain a number between `0` (or `"none"`) and `3` (or `"all"`), or `TRUE` (= `2`) or `FALSE` (= `0`) to indicate whether the input should be checked for less probable results, see *Details* #' @param keep_synonyms a [logical] to indicate if old, previously valid taxonomic names must be preserved and not be corrected to currently accepted names. The default is `FALSE`, which will return a note if old taxonomic names were processed. The default can be set with `options(AMR_keep_synonyms = TRUE)` or `options(AMR_keep_synonyms = FALSE)`. #' @param reference_df a [data.frame] to be used for extra reference when translating `x` to a valid [`mo`]. See [set_mo_source()] and [get_mo_source()] to automate the usage of your own codes (e.g. used in your analysis or organisation). -#' @param ignore_pattern a regular expression (case-insensitive) of which all matches in `x` must return `NA`. This can be convenient to exclude known non-relevant input and can also be set with the option `AMR_ignore_pattern`, e.g. `options(AMR_ignore_pattern = "(not reported|contaminated flora)")`. +#' @param ignore_pattern a [regular expression][base::regex] (case-insensitive) of which all matches in `x` must return `NA`. This can be convenient to exclude known non-relevant input and can also be set with the option `AMR_ignore_pattern`, e.g. `options(AMR_ignore_pattern = "(not reported|contaminated flora)")`. +#' @param remove_from_input a [regular expression][base::regex] (case-insensitive) to clean the input of `x`. Everything matched in `x` will be removed. At default, this is the outcome of [mo_cleaning_regex()], which removes texts between brackets and texts such as "species" and "serovar". #' @param language language to translate text like "no growth", which defaults to the system language (see [get_AMR_locale()]) #' @param info a [logical] to indicate if a progress bar should be printed if more than 25 items are to be coerced, defaults to `TRUE` only in interactive mode #' @param ... other arguments passed on to functions #' @rdname as.mo #' @aliases mo -#' @keywords mo Becker becker Lancefield lancefield guess #' @details -#' ### General Info -#' #' A microorganism (MO) code from this package (class: [`mo`]) is human readable and typically looks like these examples: #' ``` #' Code Full name @@ -60,39 +57,24 @@ #' B_KLBSL_PNMN_RHNS Klebsiella pneumoniae rhinoscleromatis #' | | | | #' | | | | -#' | | | \---> subspecies, a 4-5 letter acronym -#' | | \----> species, a 4-5 letter acronym -#' | \----> genus, a 5-7 letter acronym +#' | | | \---> subspecies, a 3-5 letter acronym +#' | | \----> species, a 3-6 letter acronym +#' | \----> genus, a 4-8 letter acronym #' \----> taxonomic kingdom: A (Archaea), AN (Animalia), B (Bacteria), #' F (Fungi), PL (Plantae), P (Protozoa) #' ``` #' -#' Values that cannot be coerced will be considered 'unknown' and will get the MO code `UNKNOWN`. +#' Values that cannot be coerced will be considered 'unknown' and will be returned as the MO code `UNKNOWN` with a warning. #' #' Use the [`mo_*`][mo_property()] functions to get properties based on the returned code, see *Examples*. -#' -#' The algorithm uses data from the List of Prokaryotic names with Standing in Nomenclature (LPSN) and the Global Biodiversity Information Facility (GBIF) (see [microorganisms]). -#' -#' The [as.mo()] function uses several coercion rules for fast and logical results. It assesses the input matching criteria in the following order: -#' -#' 1. Human pathogenic prevalence: the function starts with more prevalent microorganisms, followed by less prevalent ones; -#' 2. Taxonomic kingdom: the function starts with determining Bacteria, then Fungi, then Protozoa, then others; -#' 3. Breakdown of input values to identify possible matches. -#' -#' This will lead to the effect that e.g. `"E. coli"` (a microorganism highly prevalent in humans) will return the microbial ID of *Escherichia coli* and not *Entamoeba coli* (a microorganism less prevalent in humans), although the latter would alphabetically come first. +#' +#' The [as.mo()] function uses a novel [matching score algorithm][mo_matching_score()] (see *Matching Score for Microorganisms* below) to match input against the [available microbial taxonomy][microoganisms] in this package. This will lead to the effect that e.g. `"E. coli"` (a microorganism highly prevalent in humans) will return the microbial ID of *Escherichia coli* and not *Entamoeba coli* (a microorganism less prevalent in humans), although the latter would alphabetically come first. The algorithm uses data from the List of Prokaryotic names with Standing in Nomenclature (LPSN) and the Global Biodiversity Information Facility (GBIF) (see [microorganisms]). #' #' ### Coping with Uncertain Results #' -#' Users can control the coercion rules by setting the `allow_uncertain` argument in the [as.mo()] function. The following values or levels can be used: +#' Results of non-exact taxonomic input are based on their [matching score][mo_matching_score()]. The lowest allowed score can be set with the `minimum_matching_score` argument. At default this will be determined based on the character length of the input, and the [taxonomic kingdom][microorganisms] and [human pathogenicity][mo_matching_score()] of the taxonomic outcome. If values are matched with uncertainty, a message will be shown to suggest the user to evaluate the results with [mo_uncertainties()], which returns a [data.frame] with all specifications. #' -#' - `0`: no additional rules are applied; -#' - `1`: allow previously accepted (but now invalid) taxonomic names -#' - `2`: allow all of `1`, strip values between brackets, inverse the words of the input, strip off text elements from the end keeping at least two elements; -#' - `3`: allow all of level `1` and `2`, strip off text elements from the end, allow any part of a taxonomic name; -#' - `TRUE` (default): equivalent to `2`; -#' - `FALSE`: equivalent to `0``. -#' -#' The default is `allow_uncertain = TRUE`, which is equal to uncertainty level 2. Using `allow_uncertain = FALSE` is equal to uncertainty level 0 and will skip all rules. You can also use e.g. `as.mo(..., allow_uncertain = 1)` to only allow up to level 1 uncertainty. +#' To increase the quality of matching, the `remove_from_input` argument can be used to clean the input (i.e., `x`). This must be a [regular expression][base::regex] that matches parts of the input that should be removed before the input is matched against the [available microbial taxonomy][microoganisms]. It will be matched Perl-compatible and case-insensitive. The default value of `remove_from_input` is the outcome of the helper function [mo_cleaning_regex()]. #' #' There are three helper functions that can be run after using the [as.mo()] function: #' - Use [mo_uncertainties()] to get a [data.frame] that prints in a pretty format with all taxonomic names that were guessed. The output contains the matching score for all matches (see *Matching Score for Microorganisms* below). @@ -165,17 +147,16 @@ as.mo <- function(x, Becker = FALSE, Lancefield = FALSE, minimum_matching_score = NULL, - allow_uncertain = TRUE, keep_synonyms = getOption("AMR_keep_synonyms", FALSE), reference_df = get_mo_source(), ignore_pattern = getOption("AMR_ignore_pattern", NULL), + remove_from_input = mo_cleaning_regex(), language = get_AMR_locale(), info = interactive(), ...) { meet_criteria(x, allow_class = c("mo", "data.frame", "list", "character", "numeric", "integer", "factor"), allow_NA = TRUE) meet_criteria(Becker, allow_class = c("logical", "character"), has_length = 1) meet_criteria(Lancefield, allow_class = c("logical", "character"), has_length = 1) - meet_criteria(allow_uncertain, allow_class = c("logical", "numeric", "integer"), has_length = 1) meet_criteria(keep_synonyms, allow_class = "logical", has_length = 1) meet_criteria(minimum_matching_score, allow_class = c("numeric", "integer"), has_length = 1, allow_NULL = TRUE) meet_criteria(reference_df, allow_class = "data.frame", allow_NULL = TRUE) @@ -183,16 +164,7 @@ as.mo <- function(x, language <- validate_language(language) meet_criteria(info, allow_class = "logical", has_length = 1) - # set the microorganisms data set to use for all lookup - mo_data <- MO_lookup - - allow_uncertain <- translate_allow_uncertain(allow_uncertain) - if (allow_uncertain < 1) { - # do not allow old names - mo_data <- mo_data[which(mo_data$status == "accepted"), , drop = FALSE] - } - - if (tryCatch(all(x %in% c(mo_data$mo, NA)) && + if (tryCatch(all(x %in% c(MO_lookup$mo, NA)) && isFALSE(Becker) && isFALSE(Lancefield), error = function(e) FALSE)) { # don't look into valid MO codes, just return them @@ -221,9 +193,9 @@ as.mo <- function(x, out[x %in% reference_df[[1]]] <- reference_df[[2]][match(x[x %in% reference_df[[1]]], reference_df[[1]])] } # From MO code ---- - out[is.na(out) & x %in% mo_data$mo] <- x[is.na(out) & x %in% mo_data$mo] + out[is.na(out) & x %in% MO_lookup$mo] <- x[is.na(out) & x %in% MO_lookup$mo] # From full name ---- - out[is.na(out) & x %in% mo_data$fullname] <- mo_data$mo[match(x[is.na(out) & x %in% mo_data$fullname], mo_data$fullname)] + out[is.na(out) & x %in% MO_lookup$fullname] <- MO_lookup$mo[match(x[is.na(out) & x %in% MO_lookup$fullname], MO_lookup$fullname)] # From known codes ---- out[is.na(out) & x %in% AMR::microorganisms.codes$code] <- AMR::microorganisms.codes$mo[match(x[is.na(out) & x %in% AMR::microorganisms.codes$code], AMR::microorganisms.codes$code)] # From SNOMED ---- @@ -236,7 +208,7 @@ as.mo <- function(x, out[is.na(out)] <- convert_colloquial_input(x[is.na(out)]) # From previous hits in this session ---- old <- out - out[is.na(out) & x %in% pkg_env$mo_previously_coerced$x] <- pkg_env$mo_previously_coerced$mo[match(x[is.na(out) & x %in% pkg_env$mo_previously_coerced$x], pkg_env$mo_previously_coerced$x)] + out[is.na(out) & paste(x, minimum_matching_score) %in% pkg_env$mo_previously_coerced$x] <- pkg_env$mo_previously_coerced$mo[match(paste(x, minimum_matching_score)[is.na(out) & paste(x, minimum_matching_score) %in% pkg_env$mo_previously_coerced$x], pkg_env$mo_previously_coerced$x)] new <- out if (isTRUE(info) && message_not_thrown_before("as.mo", old, new, entire_session = TRUE) && any(is.na(old) & !is.na(new), na.rm = TRUE)) { message_( @@ -266,20 +238,17 @@ as.mo <- function(x, x_coerced <- vapply(FUN.VALUE = character(1), x_unique, function(x_search) { progress$tick() + print(x_search) + # some required cleaning steps x_out <- trimws2(x_search) - x_out <- gsub("[^A-Za-z-]+", " ", x_out, perl = TRUE) - x_out <- gsub(" +", " ", x_out, perl = TRUE) - x_out <- gsub("(^| )(e?spp|e?ssp|e?ss|e?sp|e?subsp|sube?species|biovar|biotype|serovar|e?species)( |$)", "", x_out, ignore.case = TRUE, perl = TRUE) + # this applies the `remove_from_input` argument, which defaults to mo_cleaning_regex() + x_out <- gsub(remove_from_input, " ", x_out, ignore.case = TRUE, perl = TRUE) + x_out <- trimws2(gsub(" +", " ", x_out, perl = TRUE)) x_search_cleaned <- x_out x_out <- tolower(x_out) - if (allow_uncertain == 2) { - - } - if (allow_uncertain == 3) { - - } + print(x_out) # take out the parts, split by space x_parts <- strsplit(gsub("-", " ", x_out, fixed = TRUE), " ", fixed = TRUE)[[1]] @@ -287,16 +256,16 @@ as.mo <- function(x, # do a pre-match on first character (and if it contains a space, first chars of first two terms) if (length(x_parts) %in% c(2, 3)) { # for genus + species + subspecies - filtr <- which(mo_data$full_first == substr(x_parts[1], 1, 1) & mo_data$species_first == substr(x_parts[2], 1, 1)) + filtr <- which(MO_lookup$full_first == substr(x_parts[1], 1, 1) & MO_lookup$species_first == substr(x_parts[2], 1, 1)) } else if (length(x_parts) > 3) { first_chars <- paste0("(^| )", "[", paste(substr(x_parts, 1, 1), collapse = ""), "]") - filtr <- which(mo_data$full_first %like_case% first_chars) + filtr <- which(MO_lookup$full_first %like_case% first_chars) } else if (nchar(x_out) == 4) { # no space and 4 characters - probably a code such as STAU or ESCO! if (isTRUE(info)) { message_("Input \"", x_search, "\" is assumed to be a microorganism code - trying to match on ", vector_and(c(substr(x_out, 1, 2), substr(x_out, 3, 4)), sort = FALSE)) } - filtr <- which(mo_data$fullname_lower %like_case% paste0("(^| )", substr(x_out, 1, 2), ".* ", substr(x_out, 3, 4))) + filtr <- which(MO_lookup$fullname_lower %like_case% paste0("(^| )", substr(x_out, 1, 2), ".* ", substr(x_out, 3, 4))) } else if (nchar(x_out) <= 6) { # no space and 5-6 characters - probably a code such as STAAUR or ESCCOL! first_part <- paste0(substr(x_out, 1, 2), "[a-z]*", substr(x_out, 3, 3)) @@ -304,20 +273,20 @@ as.mo <- function(x, if (isTRUE(info)) { message_("Input \"", x_search, "\" is assumed to be a microorganism code - trying to match on ", vector_and(c(gsub("[a-z]*", "(...)", first_part, fixed = TRUE), second_part), sort = FALSE)) } - filtr <- which(mo_data$fullname_lower %like_case% paste0("(^| )", first_part, ".* ", second_part)) + filtr <- which(MO_lookup$fullname_lower %like_case% paste0("(^| )", first_part, ".* ", second_part)) } else { - filtr <- which(mo_data$full_first == substr(x_out, 1, 1)) + filtr <- which(MO_lookup$full_first == substr(x_out, 1, 1)) } if (length(filtr) == 0) { - mo_to_search <- mo_data$fullname + mo_to_search <- MO_lookup$fullname } else { - mo_to_search <- mo_data$fullname[filtr] + mo_to_search <- MO_lookup$fullname[filtr] } pkg_env$mo_to_search <- mo_to_search # determine the matching score on the original search value m <- mo_matching_score(x = x_search_cleaned, n = mo_to_search) if (is.null(minimum_matching_score)) { - minimum_matching_score_current <- min(0.7, min(10, nchar(x_search_cleaned)) * 0.08) + minimum_matching_score_current <- min(0.6, min(10, nchar(x_search_cleaned)) * 0.08) # correct back for prevalence minimum_matching_score_current <- minimum_matching_score_current / MO_lookup$prevalence[match(mo_to_search, MO_lookup$fullname)] # correct back for kingdom @@ -325,7 +294,6 @@ as.mo <- function(x, } else { minimum_matching_score_current <- minimum_matching_score } - m[m < minimum_matching_score_current] <- NA_real_ top_hits <- mo_to_search[order(m, decreasing = TRUE, na.last = NA)] # na.last = NA will remove the NAs @@ -349,7 +317,7 @@ as.mo <- function(x, # save to package env to save time for next time pkg_env$mo_previously_coerced <- unique(rbind(pkg_env$mo_previously_coerced, data.frame( - x = paste(x_search, min(minimum_matching_score_current, na.rm = TRUE)), + x = paste(x_search, minimum_matching_score), mo = result_mo, stringsAsFactors = FALSE ), @@ -728,7 +696,7 @@ print.mo_uncertainties <- function(x, ...) { return(invisible(NULL)) } - cat(word_wrap("Matching scores are based on pathogenicity in humans and the resemblance between the input and the full taxonomic name. See `?mo_matching_score`.\n\n", add_fn = font_blue)) + cat(word_wrap("Matching scores are based on the resemblance between the input and the full taxonomic name, and the pathogenicity in humans. See `?mo_matching_score`.\n\n", add_fn = font_blue)) if (has_colour()) { cat(word_wrap("Colour keys: ", font_red_bg(" 0.000-0.499 "), @@ -869,6 +837,18 @@ mo_reset_session <- function() { } } +#' @rdname as.mo +#' @export +mo_cleaning_regex <- function() { + paste0( + "(", + "[^A-Za-z- \\(\\)\\[\\]{}]+", + "|", + "([({]|\\[).+([})]|\\])", + "|", + "(^| )(e?spp|e?ssp|e?ss|e?sp|e?subsp|sube?species|biovar|biotype|serovar|e?species)( |$))") +} + nr2char <- function(x) { if (x %in% c(1:10)) { v <- c( @@ -881,22 +861,6 @@ nr2char <- function(x) { } } -translate_allow_uncertain <- function(allow_uncertain) { - if (isTRUE(allow_uncertain)) { - # default to uncertainty level 2 - allow_uncertain <- 2 - } else { - allow_uncertain[tolower(allow_uncertain) == "none"] <- 0 - allow_uncertain[tolower(allow_uncertain) == "all"] <- 3 - allow_uncertain <- as.integer(allow_uncertain) - stop_ifnot(allow_uncertain %in% c(0:3), - '`allow_uncertain` must be a number between 0 (or "none") and 3 (or "all"), or TRUE (= 2) or FALSE (= 0)', - call = FALSE - ) - } - allow_uncertain -} - get_mo_uncertainties <- function() { remember <- list(uncertainties = pkg_env$mo_uncertainties) # empty them, otherwise e.g. mo_shortname("Chlamydophila psittaci") will give 3 notes diff --git a/R/mo_matching_score.R b/R/mo_matching_score.R index 63e0a5fe..ba339c1e 100755 --- a/R/mo_matching_score.R +++ b/R/mo_matching_score.R @@ -33,6 +33,7 @@ #' @author Dr. Matthijs Berends #' @param x Any user input value(s) #' @param n A full taxonomic name, that exists in [`microorganisms$fullname`][microorganisms] +#' @note This algorithm was described in: Berends MS *et al.* (2022). **AMR: An R Package for Working with Antimicrobial Resistance Data**. *Journal of Statistical Software*, 104(3), 1-31; \doi{10.18637/jss.v104.i03}. #' @section Matching Score for Microorganisms: #' With ambiguous user input in [as.mo()] and all the [`mo_*`][mo_property()] functions, the returned results are chosen based on their matching score using [mo_matching_score()]. This matching score \eqn{m}, is calculated as: #' @@ -43,7 +44,7 @@ #' * \ifelse{html}{\out{x is the user input;}}{\eqn{x} is the user input;} #' * \ifelse{html}{\out{n is a taxonomic name (genus, species, and subspecies);}}{\eqn{n} is a taxonomic name (genus, species, and subspecies);} #' * \ifelse{html}{\out{ln is the length of n;}}{l_n is the length of \eqn{n};} -#' * \ifelse{html}{\out{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change x into n;}}{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change \eqn{x} into \eqn{n};} +#' * \ifelse{html}{\out{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change x into n;}}{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change \eqn{x} into \eqn{n};} #' * \ifelse{html}{\out{pn is the human pathogenic prevalence group of n, as described below;}}{p_n is the human pathogenic prevalence group of \eqn{n}, as described below;} #' * \ifelse{html}{\out{kn is the taxonomic kingdom of n, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.}}{l_n is the taxonomic kingdom of \eqn{n}, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.} #' @@ -86,19 +87,19 @@ mo_matching_score <- function(x, n) { if (length(x) == 1) { x <- rep(x, length(n)) } - + # length of fullname l_n <- nchar(n) lev <- double(length = length(x)) l_n.lev <- double(length = length(x)) - lev <- unlist(Map( - f = utils::adist, - x, - n, - ignore.case = FALSE, - USE.NAMES = FALSE, - fixed = TRUE - )) + lev <- unlist(Map(f = function(a, b) { + as.double(utils::adist(a, b, + ignore.case = FALSE, + fixed = TRUE, + costs = c(insertions = 1, deletions = 2, substitutions = 2), + counts = FALSE)) + }, x, n, USE.NAMES = FALSE)) + l_n.lev[l_n < lev] <- l_n[l_n < lev] l_n.lev[lev < l_n] <- lev[lev < l_n] l_n.lev[lev == l_n] <- lev[lev == l_n] diff --git a/R/mo_property.R b/R/mo_property.R index 11d5230e..f57371a6 100755 --- a/R/mo_property.R +++ b/R/mo_property.R @@ -782,7 +782,7 @@ mo_validate <- function(x, property, language, keep_synonyms = keep_synonyms, .. # try to catch an error when inputting an invalid argument # so the 'call.' can be set to FALSE - tryCatch(x[1L] %in% AMR::microorganisms[1, property, drop = TRUE], + tryCatch(x[1L] %in% unlist(AMR::microorganisms[1, property, drop = TRUE]), error = function(e) stop(e$message, call. = FALSE) ) @@ -802,7 +802,7 @@ mo_validate <- function(x, property, language, keep_synonyms = keep_synonyms, .. if (all(x %in% c(mo_data_check$mo, NA)) && !has_Becker_or_Lancefield) { # do nothing, just don't run the other if-else's - } else if (all(x %in% c(mo_data_check[[property]], NA)) && !has_Becker_or_Lancefield) { + } else if (all(x %in% c(unlist(mo_data_check[[property]]), NA)) && !has_Becker_or_Lancefield) { # no need to do anything, just return it return(x) } else { diff --git a/inst/tinytest/test-mo.R b/inst/tinytest/test-mo.R index d49d9370..7106ada4 100644 --- a/inst/tinytest/test-mo.R +++ b/inst/tinytest/test-mo.R @@ -232,14 +232,6 @@ expect_equal(suppressMessages(as.character(as.mo("Escherichia blattae"))), "B_SH expect_equal(suppressMessages(as.character(as.mo(c("E. coli", "Chlamydo psittaci")))), c("B_ESCHR_COLI", "B_CHLMY_PSTT")) expect_equal(suppressMessages(mo_name("eubcom")), "Clostridium combesii") -# check uncertain names -expect_equal(suppressMessages(as.character(as.mo("staaur extratest", allow_uncertain = TRUE))), "B_STPHY_AURS") -expect_equal(suppressWarnings(as.character(as.mo("staaur extratest", allow_uncertain = FALSE))), "UNKNOWN") -expect_message(as.mo("e coli extra_text", allow_uncertain = TRUE)) -expect_equal(suppressMessages(as.character(as.mo("unexisting aureus", allow_uncertain = 3))), "B_STPHY_AURS") -expect_equal(suppressMessages(as.character(as.mo("unexisting staphy", allow_uncertain = 3))), "B_STPHY_COPS") -expect_equal(suppressMessages(as.character(as.mo(c("s aure THISISATEST", "Staphylococcus aureus unexisting"), allow_uncertain = 3))), c("B_STPHY_AURS_AURS", "B_STPHY_AURS_AURS")) - # predefined reference_df expect_equal( as.character(as.mo("TestingOwnID", @@ -311,11 +303,6 @@ expect_equal( rep("UNKNOWN", 3) ) -expect_error(translate_allow_uncertain(5)) - -# debug mode -expect_stdout(print(suppressMessages(suppressWarnings(as.mo("kshgcjkhsdgkshjdfsfvsdfv", debug = TRUE, allow_uncertain = 3))))) - # ..coccus expect_equal( as.character(as.mo(c("meningococ", "gonococ", "pneumococ"))), diff --git a/man/as.mo.Rd b/man/as.mo.Rd index dedbabfc..4e8f4b1e 100644 --- a/man/as.mo.Rd +++ b/man/as.mo.Rd @@ -8,6 +8,7 @@ \alias{mo_uncertainties} \alias{mo_renamed} \alias{mo_reset_session} +\alias{mo_cleaning_regex} \title{Transform Input to a Microorganism Code} \usage{ as.mo( @@ -15,10 +16,10 @@ as.mo( Becker = FALSE, Lancefield = FALSE, minimum_matching_score = NULL, - allow_uncertain = TRUE, keep_synonyms = getOption("AMR_keep_synonyms", FALSE), reference_df = get_mo_source(), ignore_pattern = getOption("AMR_ignore_pattern", NULL), + remove_from_input = mo_cleaning_regex(), language = get_AMR_locale(), info = interactive(), ... @@ -33,6 +34,8 @@ mo_uncertainties() mo_renamed() mo_reset_session() + +mo_cleaning_regex() } \arguments{ \item{x}{a \link{character} vector or a \link{data.frame} with one or two columns} @@ -47,13 +50,13 @@ This excludes enterococci at default (who are in group D), use \code{Lancefield \item{minimum_matching_score}{a numeric value to set as the lower limit for the \link[=mo_matching_score]{MO matching score}. When left blank, this will be determined automatically based on the character length of \code{x}, its \link[=microorganisms]{taxonomic kingdom} and \link[=mo_matching_score]{human pathogenicity}.} -\item{allow_uncertain}{a number between \code{0} (or \code{"none"}) and \code{3} (or \code{"all"}), or \code{TRUE} (= \code{2}) or \code{FALSE} (= \code{0}) to indicate whether the input should be checked for less probable results, see \emph{Details}} - \item{keep_synonyms}{a \link{logical} to indicate if old, previously valid taxonomic names must be preserved and not be corrected to currently accepted names. The default is \code{FALSE}, which will return a note if old taxonomic names were processed. The default can be set with \code{options(AMR_keep_synonyms = TRUE)} or \code{options(AMR_keep_synonyms = FALSE)}.} \item{reference_df}{a \link{data.frame} to be used for extra reference when translating \code{x} to a valid \code{\link{mo}}. See \code{\link[=set_mo_source]{set_mo_source()}} and \code{\link[=get_mo_source]{get_mo_source()}} to automate the usage of your own codes (e.g. used in your analysis or organisation).} -\item{ignore_pattern}{a regular expression (case-insensitive) of which all matches in \code{x} must return \code{NA}. This can be convenient to exclude known non-relevant input and can also be set with the option \code{AMR_ignore_pattern}, e.g. \code{options(AMR_ignore_pattern = "(not reported|contaminated flora)")}.} +\item{ignore_pattern}{a \link[base:regex]{regular expression} (case-insensitive) of which all matches in \code{x} must return \code{NA}. This can be convenient to exclude known non-relevant input and can also be set with the option \code{AMR_ignore_pattern}, e.g. \code{options(AMR_ignore_pattern = "(not reported|contaminated flora)")}.} + +\item{remove_from_input}{a \link[base:regex]{regular expression} (case-insensitive) to clean the input of \code{x}. Everything matched in \code{x} will be removed. At default, this is the outcome of \code{\link[=mo_cleaning_regex]{mo_cleaning_regex()}}, which removes texts between brackets and texts such as "species" and "serovar".} \item{language}{language to translate text like "no growth", which defaults to the system language (see \code{\link[=get_AMR_locale]{get_AMR_locale()}})} @@ -68,8 +71,6 @@ A \link{character} \link{vector} with additional class \code{\link{mo}} Use this function to determine a valid microorganism code (\code{\link{mo}}). Determination is done using intelligent rules and the complete taxonomic kingdoms Animalia, Archaea, Bacteria and Protozoa, and most microbial species from the kingdom Fungi (see \emph{Source}). The input can be almost anything: a full name (like \code{"Staphylococcus aureus"}), an abbreviated name (such as \code{"S. aureus"}), an abbreviation known in the field (such as \code{"MRSA"}), or just a genus. See \emph{Examples}. } \details{ -\subsection{General Info}{ - A microorganism (MO) code from this package (class: \code{\link{mo}}) is human readable and typically looks like these examples: \if{html}{\out{
}}\preformatted{ Code Full name @@ -79,42 +80,23 @@ A microorganism (MO) code from this package (class: \code{\link{mo}}) is human r B_KLBSL_PNMN_RHNS Klebsiella pneumoniae rhinoscleromatis | | | | | | | | - | | | \\---> subspecies, a 4-5 letter acronym - | | \\----> species, a 4-5 letter acronym - | \\----> genus, a 5-7 letter acronym + | | | \\---> subspecies, a 3-5 letter acronym + | | \\----> species, a 3-6 letter acronym + | \\----> genus, a 4-8 letter acronym \\----> taxonomic kingdom: A (Archaea), AN (Animalia), B (Bacteria), F (Fungi), PL (Plantae), P (Protozoa) }\if{html}{\out{
}} -Values that cannot be coerced will be considered 'unknown' and will get the MO code \code{UNKNOWN}. +Values that cannot be coerced will be considered 'unknown' and will be returned as the MO code \code{UNKNOWN} with a warning. Use the \code{\link[=mo_property]{mo_*}} functions to get properties based on the returned code, see \emph{Examples}. -The algorithm uses data from the List of Prokaryotic names with Standing in Nomenclature (LPSN) and the Global Biodiversity Information Facility (GBIF) (see \link{microorganisms}). - -The \code{\link[=as.mo]{as.mo()}} function uses several coercion rules for fast and logical results. It assesses the input matching criteria in the following order: -\enumerate{ -\item Human pathogenic prevalence: the function starts with more prevalent microorganisms, followed by less prevalent ones; -\item Taxonomic kingdom: the function starts with determining Bacteria, then Fungi, then Protozoa, then others; -\item Breakdown of input values to identify possible matches. -} - -This will lead to the effect that e.g. \code{"E. coli"} (a microorganism highly prevalent in humans) will return the microbial ID of \emph{Escherichia coli} and not \emph{Entamoeba coli} (a microorganism less prevalent in humans), although the latter would alphabetically come first. -} - +The \code{\link[=as.mo]{as.mo()}} function uses a novel \link[=mo_matching_score]{matching score algorithm} (see \emph{Matching Score for Microorganisms} below) to match input against the \link[=microoganisms]{available microbial taxonomy} in this package. This will lead to the effect that e.g. \code{"E. coli"} (a microorganism highly prevalent in humans) will return the microbial ID of \emph{Escherichia coli} and not \emph{Entamoeba coli} (a microorganism less prevalent in humans), although the latter would alphabetically come first. The algorithm uses data from the List of Prokaryotic names with Standing in Nomenclature (LPSN) and the Global Biodiversity Information Facility (GBIF) (see \link{microorganisms}). \subsection{Coping with Uncertain Results}{ -Users can control the coercion rules by setting the \code{allow_uncertain} argument in the \code{\link[=as.mo]{as.mo()}} function. The following values or levels can be used: -\itemize{ -\item \code{0}: no additional rules are applied; -\item \code{1}: allow previously accepted (but now invalid) taxonomic names -\item \code{2}: allow all of \code{1}, strip values between brackets, inverse the words of the input, strip off text elements from the end keeping at least two elements; -\item \code{3}: allow all of level \code{1} and \code{2}, strip off text elements from the end, allow any part of a taxonomic name; -\item \code{TRUE} (default): equivalent to \code{2}; -\item \code{FALSE}: equivalent to `0``. -} +Results of non-exact taxonomic input are based on their \link[=mo_matching_score]{matching score}. The lowest allowed score can be set with the \code{minimum_matching_score} argument. At default this will be determined based on the character length of the input, and the \link[=microorganisms]{taxonomic kingdom} and \link[=mo_matching_score]{human pathogenicity} of the taxonomic outcome. If values are matched with uncertainty, a message will be shown to suggest the user to evaluate the results with \code{\link[=mo_uncertainties]{mo_uncertainties()}}, which returns a \link{data.frame} with all specifications. -The default is \code{allow_uncertain = TRUE}, which is equal to uncertainty level 2. Using \code{allow_uncertain = FALSE} is equal to uncertainty level 0 and will skip all rules. You can also use e.g. \code{as.mo(..., allow_uncertain = 1)} to only allow up to level 1 uncertainty. +To increase the quality of matching, the \code{remove_from_input} argument can be used to clean the input (i.e., \code{x}). This must be a \link[base:regex]{regular expression} that matches parts of the input that should be removed before the input is matched against the \link[=microoganisms]{available microbial taxonomy}. It will be matched Perl-compatible and case-insensitive. The default value of \code{remove_from_input} is the outcome of the helper function \code{\link[=mo_cleaning_regex]{mo_cleaning_regex()}}. There are three helper functions that can be run after using the \code{\link[=as.mo]{as.mo()}} function: \itemize{ @@ -155,7 +137,7 @@ where: \item \ifelse{html}{\out{x is the user input;}}{\eqn{x} is the user input;} \item \ifelse{html}{\out{n is a taxonomic name (genus, species, and subspecies);}}{\eqn{n} is a taxonomic name (genus, species, and subspecies);} \item \ifelse{html}{\out{ln is the length of n;}}{l_n is the length of \eqn{n};} -\item \ifelse{html}{\out{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change x into n;}}{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change \eqn{x} into \eqn{n};} +\item \ifelse{html}{\out{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change x into n;}}{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change \eqn{x} into \eqn{n};} \item \ifelse{html}{\out{pn is the human pathogenic prevalence group of n, as described below;}}{p_n is the human pathogenic prevalence group of \eqn{n}, as described below;} \item \ifelse{html}{\out{kn is the taxonomic kingdom of n, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.}}{l_n is the taxonomic kingdom of \eqn{n}, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.} } @@ -224,9 +206,3 @@ mo_is_intrinsic_resistant("ESCCOL", ab = "vanco") The \code{\link[=mo_property]{mo_*}} functions (such as \code{\link[=mo_genus]{mo_genus()}}, \code{\link[=mo_gramstain]{mo_gramstain()}}) to get properties based on the returned code. } -\keyword{Becker} -\keyword{Lancefield} -\keyword{becker} -\keyword{guess} -\keyword{lancefield} -\keyword{mo} diff --git a/man/mo_matching_score.Rd b/man/mo_matching_score.Rd index edb1f5f4..91a324a9 100644 --- a/man/mo_matching_score.Rd +++ b/man/mo_matching_score.Rd @@ -14,6 +14,9 @@ mo_matching_score(x, n) \description{ This algorithm is used by \code{\link[=as.mo]{as.mo()}} and all the \code{\link[=mo_property]{mo_*}} functions to determine the most probable match of taxonomic records based on user input. } +\note{ +This algorithm was described in: Berends MS \emph{et al.} (2022). \strong{AMR: An R Package for Working with Antimicrobial Resistance Data}. \emph{Journal of Statistical Software}, 104(3), 1-31; \doi{10.18637/jss.v104.i03}. +} \section{Matching Score for Microorganisms}{ With ambiguous user input in \code{\link[=as.mo]{as.mo()}} and all the \code{\link[=mo_property]{mo_*}} functions, the returned results are chosen based on their matching score using \code{\link[=mo_matching_score]{mo_matching_score()}}. This matching score \eqn{m}, is calculated as: @@ -25,7 +28,7 @@ where: \item \ifelse{html}{\out{x is the user input;}}{\eqn{x} is the user input;} \item \ifelse{html}{\out{n is a taxonomic name (genus, species, and subspecies);}}{\eqn{n} is a taxonomic name (genus, species, and subspecies);} \item \ifelse{html}{\out{ln is the length of n;}}{l_n is the length of \eqn{n};} -\item \ifelse{html}{\out{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change x into n;}}{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change \eqn{x} into \eqn{n};} +\item \ifelse{html}{\out{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change x into n;}}{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change \eqn{x} into \eqn{n};} \item \ifelse{html}{\out{pn is the human pathogenic prevalence group of n, as described below;}}{p_n is the human pathogenic prevalence group of \eqn{n}, as described below;} \item \ifelse{html}{\out{kn is the taxonomic kingdom of n, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.}}{l_n is the taxonomic kingdom of \eqn{n}, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.} } diff --git a/man/mo_property.Rd b/man/mo_property.Rd index 8c7cef81..8adba27b 100644 --- a/man/mo_property.Rd +++ b/man/mo_property.Rd @@ -316,7 +316,7 @@ where: \item \ifelse{html}{\out{x is the user input;}}{\eqn{x} is the user input;} \item \ifelse{html}{\out{n is a taxonomic name (genus, species, and subspecies);}}{\eqn{n} is a taxonomic name (genus, species, and subspecies);} \item \ifelse{html}{\out{ln is the length of n;}}{l_n is the length of \eqn{n};} -\item \ifelse{html}{\out{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change x into n;}}{lev is the Levenshtein distance function, which counts any insertion, deletion and substitution as 1 that is needed to change \eqn{x} into \eqn{n};} +\item \ifelse{html}{\out{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change x into n;}}{lev is the Levenshtein distance function (counting any insertion as 1, and any deletion or substitution as 2) that is needed to change \eqn{x} into \eqn{n};} \item \ifelse{html}{\out{pn is the human pathogenic prevalence group of n, as described below;}}{p_n is the human pathogenic prevalence group of \eqn{n}, as described below;} \item \ifelse{html}{\out{kn is the taxonomic kingdom of n, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.}}{l_n is the taxonomic kingdom of \eqn{n}, set as Bacteria = 1, Fungi = 2, Protozoa = 3, Archaea = 4, others = 5.} }