data_1st %>%
group_by(hospital) %>%
@@ -984,19 +984,19 @@ Longest: 24
Hospital A
-0.4661749
+0.4814277
Hospital B
-0.4820906
+0.4811252
Hospital C
-0.4847862
+0.4843815
Hospital D
-0.4790875
+0.4610721
@@ -1014,23 +1014,23 @@ Longest: 24
Hospital A
-0.4661749
-4745
+0.4814277
+4819
Hospital B
-0.4820906
-5472
+0.4811252
+5510
Hospital C
-0.4847862
-2432
+0.4843815
+2401
Hospital D
-0.4790875
-3156
+0.4610721
+3134
@@ -1050,27 +1050,27 @@ Longest: 24
Escherichia
-0.7332054
-0.9033909
-0.9754319
+0.7180533
+0.8983310
+0.9704421
Klebsiella
-0.7142857
-0.9095675
-0.9737877
+0.7189501
+0.8879641
+0.9654289
Staphylococcus
-0.7179037
-0.9162487
-0.9779338
+0.7231920
+0.9216958
+0.9790524
Streptococcus
-0.7289984
+0.7372084
0.0000000
-0.7289984
+0.7372084
diff --git a/docs/articles/AMR_files/figure-html/plot 1-1.png b/docs/articles/AMR_files/figure-html/plot 1-1.png
index 86aa274d..b87bb214 100644
Binary files a/docs/articles/AMR_files/figure-html/plot 1-1.png and b/docs/articles/AMR_files/figure-html/plot 1-1.png differ
diff --git a/docs/articles/AMR_files/figure-html/plot 3-1.png b/docs/articles/AMR_files/figure-html/plot 3-1.png
index 1f6d21fd..3a6a2c1f 100644
Binary files a/docs/articles/AMR_files/figure-html/plot 3-1.png and b/docs/articles/AMR_files/figure-html/plot 3-1.png differ
diff --git a/docs/articles/AMR_files/figure-html/plot 4-1.png b/docs/articles/AMR_files/figure-html/plot 4-1.png
index 2689fa2e..eec1e91b 100644
Binary files a/docs/articles/AMR_files/figure-html/plot 4-1.png and b/docs/articles/AMR_files/figure-html/plot 4-1.png differ
diff --git a/docs/articles/AMR_files/figure-html/plot 5-1.png b/docs/articles/AMR_files/figure-html/plot 5-1.png
index bac17cf6..dae88c56 100644
Binary files a/docs/articles/AMR_files/figure-html/plot 5-1.png and b/docs/articles/AMR_files/figure-html/plot 5-1.png differ
diff --git a/docs/articles/EUCAST.html b/docs/articles/EUCAST.html
index 908711d9..8eabbc3a 100644
--- a/docs/articles/EUCAST.html
+++ b/docs/articles/EUCAST.html
@@ -192,7 +192,7 @@
How to apply EUCAST rules
Matthijs S. Berends
- 18 February 2019
+ 20 February 2019
EUCAST.Rmd
diff --git a/docs/articles/G_test.html b/docs/articles/G_test.html
index 589abac2..304f848b 100644
--- a/docs/articles/G_test.html
+++ b/docs/articles/G_test.html
@@ -192,7 +192,7 @@
How to use the G-test
Matthijs S. Berends
- 18 February 2019
+ 20 February 2019
G_test.Rmd
diff --git a/docs/articles/WHONET.html b/docs/articles/WHONET.html
index 9d3b980e..52633355 100644
--- a/docs/articles/WHONET.html
+++ b/docs/articles/WHONET.html
@@ -192,7 +192,7 @@
How to work with WHONET data
Matthijs S. Berends
- 18 February 2019
+ 20 February 2019
WHONET.Rmd
diff --git a/docs/articles/atc_property.html b/docs/articles/atc_property.html
index 0760a382..f9bcbfe3 100644
--- a/docs/articles/atc_property.html
+++ b/docs/articles/atc_property.html
@@ -192,7 +192,7 @@
How to get properties of an antibiotic
Matthijs S. Berends
- 18 February 2019
+ 20 February 2019
atc_property.Rmd
diff --git a/docs/articles/benchmarks.html b/docs/articles/benchmarks.html
index 7fdf2257..f50800a3 100644
--- a/docs/articles/benchmarks.html
+++ b/docs/articles/benchmarks.html
@@ -192,7 +192,7 @@
Benchmarks
Matthijs S. Berends
- 14 February 2019
+ 20 February 2019
benchmarks.Rmd
@@ -201,161 +201,183 @@
-One of the most important features of this package is the complete microbial taxonomic database, supplied by ITIS (https://www.itis.gov). We created a function as.mo() that transforms any user input value to a valid microbial ID by using AI (Artificial Intelligence) and based on the taxonomic tree of ITIS.
-Using the microbenchmark package, we can review the calculation performance of this function. Its function microbenchmark() calculates different input expressions independently of each others and runs every expression 100 times.
+One of the most important features of this package is the complete microbial taxonomic database, supplied by the Catalogue of Life (http://catalogueoflife.org). We created a function as.mo() that transforms any user input value to a valid microbial ID by using AI (Artificial Intelligence) combined with the taxonomic tree of Catalogue of Life.
+Using the microbenchmark package, we can review the calculation performance of this function. Its function microbenchmark() runs different input expressions independently of each other and measures their time-to-result.
In the next test, we try to ‘coerce’ different input values for Staphylococcus aureus. The actual result is the same every time: it returns its MO code B_STPHY_AUR (B stands for Bacteria, the taxonomic kingdom).
But the calculation time differs a lot. Here, the AI effect can be reviewed best:
-benchmark <- microbenchmark(as.mo("sau"),
- as.mo("stau"),
- as.mo("staaur"),
- as.mo("S. aureus"),
- as.mo("S. aureus"),
- as.mo("STAAUR"),
- as.mo("Staphylococcus aureus"),
- as.mo("B_STPHY_AUR"))
-print(benchmark, unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# as.mo("sau") 18.983141 19.121148 19.9676944 19.1967505 19.2871260 38.635012 100
-# as.mo("stau") 37.503863 37.692049 38.9856547 37.8244335 37.9851040 57.576107 100
-# as.mo("staaur") 18.945427 19.122579 19.6392560 19.2241285 19.3536140 38.687672 100
-# as.mo("S. aureus") 15.305229 15.471103 16.3477096 15.5545630 15.6689280 36.363005 100
-# as.mo("S. aureus") 15.308232 15.469881 16.5269706 15.5506870 15.6277560 42.155292 100
-# as.mo("STAAUR") 18.984049 19.117166 19.6104597 19.2219285 19.3161095 38.638783 100
-# as.mo("Staphylococcus aureus") 8.103546 8.198285 8.6422018 8.2636915 8.3200535 27.002527 100
-# as.mo("B_STPHY_AUR") 0.156236 0.196779 0.2017926 0.2035535 0.2115505 0.241861 100
-
-par(mar = c(5, 15, 4, 2)) # set more space for left margin text (15)
-boxplot(benchmark, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, 200),
- main = expression(paste("Benchmark of ", italic("Staphylococcus aureus"))))
-
-In the table above, all measurements are shown in milliseconds (thousands of seconds), tested on a quite regular Linux server from 2007 (Core 2 Duo 2.7 GHz, 2 GB DDR2 RAM). A value of 8 milliseconds means it can determine 125 input values per second. It case of 40 milliseconds, this is only 25 input values per second. The more an input value resembles a full name, the faster the result will be found. In case of as.mo("B_STPHY_AUR"), the input is already a valid MO code, so it only almost takes no time at all (0.0002 seconds on our server).
-To achieve this speed, the as.mo function also takes into account the prevalence of human pathogenic microorganisms. The downside is of course that less prevalent microorganisms will be determined far less faster. See this example for the ID of Burkholderia nodosa (B_BRKHL_NOD):
-benchmark <- microbenchmark(as.mo("buno"),
- as.mo("burnod"),
- as.mo("B. nodosa"),
- as.mo("B. nodosa"),
- as.mo("BURNOD"),
- as.mo("Burkholderia nodosa"),
- as.mo("B_BRKHL_NOD"))
-print(benchmark, unit = "ms")
-
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# as.mo("buno") 125.141333 125.8553210 129.5727691 126.3899910 127.0954925 194.51985 100
-# as.mo("burnod") 142.300359 144.1611750 147.0642288 144.6074960 145.5243025 176.91649 100
-# as.mo("B. nodosa") 81.530132 81.9360840 83.3915418 82.1852770 82.6848870 102.63184 100
-# as.mo("B. nodosa") 81.109547 81.9836805 84.7595894 82.3437825 82.8282705 110.67036 100
-# as.mo("BURNOD") 143.163527 143.9134485 148.7192688 144.5582580 145.7489115 314.92070 100
-# as.mo("Burkholderia nodosa") 36.226325 36.5499000 37.1309929 36.6581540 36.7551985 56.25597 100
-# as.mo("B_BRKHL_NOD") 0.172509 0.3038455 0.4806591 0.3078265 0.3121215 19.16173 100
-
-boxplot(benchmark, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, 200),
- main = expression(paste("Benchmark of ", italic("Burkholderia nodosa"))))
-
-That takes up to 8 times as much time! A value of 145 milliseconds means it can only determine ~7 different input values per second. We can conclude that looking up arbitrary codes of less prevalent microorganisms is the worst way to go, in terms of calculation performance.
+S.aureus <- microbenchmark(as.mo("sau"),
+ as.mo("stau"),
+ as.mo("staaur"),
+ as.mo("S. aureus"),
+ as.mo("S. aureus"),
+ as.mo("STAAUR"),
+ as.mo("Staphylococcus aureus"),
+ as.mo("B_STPHY_AUR"),
+ times = 10)
+print(S.aureus, unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean median
+#> as.mo("sau") 42.58139 42.645368 43.3006677 42.970095
+#> as.mo("stau") 76.60094 77.168264 83.7686909 77.316642
+#> as.mo("staaur") 42.86607 42.947083 43.5035571 43.497293
+#> as.mo("S. aureus") 18.39354 18.432582 22.4304233 18.495928
+#> as.mo("S. aureus") 18.46513 18.559903 18.6640991 18.579110
+#> as.mo("STAAUR") 42.71975 42.788612 44.3280682 43.069864
+#> as.mo("Staphylococcus aureus") 11.56285 11.591419 15.9457298 11.667161
+#> as.mo("B_STPHY_AUR") 0.40487 0.450128 0.5036822 0.481417
+#> uq max neval
+#> 43.448543 45.058105 10
+#> 78.335591 127.180349 10
+#> 43.817095 44.999509 10
+#> 19.007097 56.501460 10
+#> 18.651814 19.373275 10
+#> 43.741388 54.703256 10
+#> 12.323077 50.121808 10
+#> 0.519271 0.766578 10
+In the table above, all measurements are shown in milliseconds (thousands of seconds). A value of 10 milliseconds means it can determine 100 input values per second. It case of 50 milliseconds, this is only 20 input values per second. The more an input value resembles a full name, the faster the result will be found. In case of as.mo("B_STPHY_AUR"), the input is already a valid MO code, so it only almost takes no time at all (404 millionths of seconds).
+To achieve this speed, the as.mo function also takes into account the prevalence of human pathogenic microorganisms. The downside is of course that less prevalent microorganisms will be determined far less faster. See this example for the ID of Mycoplasma leonicaptivi (B_MYCPL_LEO), a bug probably never found before in humans:
+M.leonicaptivi <- microbenchmark(as.mo("myle"),
+ as.mo("mycleo"),
+ as.mo("M. leonicaptivi"),
+ as.mo("M. leonicaptivi"),
+ as.mo("MYCLEO"),
+ as.mo("Mycoplasma leonicaptivi"),
+ as.mo("B_MYCPL_LEO"),
+ times = 10)
+print(M.leonicaptivi, unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean
+#> as.mo("myle") 112.28656 112.372601 112.751678
+#> as.mo("mycleo") 382.46812 382.757612 383.432440
+#> as.mo("M. leonicaptivi") 202.68674 203.654949 210.461303
+#> as.mo("M. leonicaptivi") 202.89759 203.440956 203.816387
+#> as.mo("MYCLEO") 382.27864 383.090895 401.904482
+#> as.mo("Mycoplasma leonicaptivi") 102.99676 103.191196 109.196394
+#> as.mo("B_MYCPL_LEO") 0.32155 0.564807 4.320068
+#> median uq max neval
+#> 112.540884 112.76874 113.76321 10
+#> 383.232219 384.05897 385.28587 10
+#> 204.255445 205.80976 242.53035 10
+#> 203.613673 203.82802 206.15038 10
+#> 386.478757 421.87837 437.26978 10
+#> 103.596136 104.65940 142.25748 10
+#> 0.593652 0.62522 37.96384 10
+That takes 6 times as much time on average! A value of 100 milliseconds means it can only determine ~10 different input values per second. We can conclude that looking up arbitrary codes of less prevalent microorganisms is the worst way to go, in terms of calculation performance:
+par(mar = c(5, 16, 4, 2)) # set more space for left margin text (16)
+
+# highest value on y axis
+max_y_axis <- max(S.aureus$time, M.leonicaptivi$time, na.rm = TRUE) / 1e6
+
+boxplot(S.aureus, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, max_y_axis),
+ main = expression(paste("Benchmark of ", italic("Staphylococcus aureus"))))
+
+
+boxplot(M.leonicaptivi, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, max_y_axis),
+ main = expression(paste("Benchmark of ", italic("Mycoplasma leonicaptivi"))))
+
To relieve this pitfall and further improve performance, two important calculations take almost no time at all: repetitive results and already precalculated results.
Repetitive results
-Repetitive results mean that unique values are present more than once. Unique values will only be calculated once by as.mo(). We will use mo_fullname() for this test - a helper function that returns the full microbial name (genus, species and possibly subspecies) and uses as.mo() internally.
-library(dplyr)
-# take 500,000 random MO codes from the septic_patients data set
-x = septic_patients %>%
- sample_n(500000, replace = TRUE) %>%
- pull(mo)
-
-# got the right length?
-length(x)
-# [1] 500000
-
-# and how many unique values do we have?
-n_distinct(x)
-# [1] 96
-
-# only 96, but distributed in 500,000 results. now let's see:
-microbenchmark(X = mo_fullname(x),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# X 114.9342 117.1076 129.6448 120.2047 131.5005 168.6371 10
-So transforming 500,000 values (!) of 96 unique values only takes 0.12 seconds (120 ms). You only lose time on your unique input values.
-Results of a tenfold - 5,000,000 values:
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# X 882.9045 901.3011 1001.677 940.3421 1168.088 1226.846 10
-Even determining the full names of 5 Million values is done within a second.
+Repetitive results mean that unique values are present more than once. Unique values will only be calculated once by as.mo(). We will use mo_fullname() for this test - a helper function that returns the full microbial name (genus, species and possibly subspecies) which uses as.mo() internally.
+library(dplyr)
+#>
+#> Attaching package: 'dplyr'
+#> The following objects are masked from 'package:stats':
+#>
+#> filter, lag
+#> The following objects are masked from 'package:base':
+#>
+#> intersect, setdiff, setequal, union
+# take 500,000 random MO codes from the septic_patients data set
+x = septic_patients %>%
+ sample_n(500000, replace = TRUE) %>%
+ pull(mo)
+
+# got the right length?
+length(x)
+#> [1] 500000
+
+# and how many unique values do we have?
+n_distinct(x)
+#> [1] 95
+
+# now let's see:
+run_it <- microbenchmark(X = mo_fullname(x),
+ times = 10)
+print(run_it, unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean median uq max neval
+#> X 435.7086 442.1682 465.5949 468.8453 477.1915 505.961 10
+So transforming 500,000 values (!) of 95 unique values only takes 0.47 seconds (468 ms). You only lose time on your unique input values.
Precalculated results
What about precalculated results? If the input is an already precalculated result of a helper function like mo_fullname(), it almost doesn’t take any time at all (see ‘C’ below):
-microbenchmark(A = mo_fullname("B_STPHY_AUR"),
- B = mo_fullname("S. aureus"),
- C = mo_fullname("Staphylococcus aureus"),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# A 11.364086 11.460537 11.5104799 11.4795330 11.524860 11.818263 10
-# B 11.976454 12.012352 12.1704592 12.0853020 12.210004 12.881737 10
-# C 0.095823 0.102528 0.1167754 0.1153785 0.132629 0.140661 10
-So going from mo_fullname("Staphylococcus aureus") to "Staphylococcus aureus" takes 0.0001 seconds - it doesn’t even start calculating if the result would be the same as the expected resulting value. That goes for all helper functions:
-microbenchmark(A = mo_species("aureus"),
- B = mo_genus("Staphylococcus"),
- C = mo_fullname("Staphylococcus aureus"),
- D = mo_family("Staphylococcaceae"),
- E = mo_order("Bacillales"),
- F = mo_class("Bacilli"),
- G = mo_phylum("Firmicutes"),
- H = mo_subkingdom("Posibacteria"),
- I = mo_kingdom("Bacteria"),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# A 0.105181 0.121314 0.1478538 0.1465265 0.166711 0.211409 10
-# B 0.132558 0.146388 0.1584278 0.1499835 0.164895 0.208477 10
-# C 0.135492 0.160355 0.2341847 0.1884665 0.348857 0.395931 10
-# D 0.109650 0.115727 0.1270481 0.1264130 0.128648 0.168317 10
-# E 0.081574 0.096940 0.0992582 0.0980915 0.101479 0.120477 10
-# F 0.081575 0.088489 0.0988463 0.0989650 0.103365 0.126482 10
-# G 0.091981 0.095333 0.1043568 0.1001530 0.111327 0.129625 10
-# H 0.092610 0.093169 0.1009135 0.0985455 0.101828 0.120406 10
-# I 0.087371 0.091213 0.1069758 0.0941815 0.109302 0.192831 10
-Of course, when running mo_phylum("Firmicutes") the function has zero knowledge about the actual microorganism, namely S. aureus. But since the result would be "Firmicutes" too, there is no point in calculating the result. And because this package ‘knows’ all phyla of all known microorganisms (according to ITIS), it can just return the initial value immediately.
+run_it <- microbenchmark(A = mo_fullname("B_STPHY_AUR"),
+ B = mo_fullname("S. aureus"),
+ C = mo_fullname("Staphylococcus aureus"),
+ times = 10)
+print(run_it, unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean median uq max neval
+#> A 38.887977 38.920313 39.3674024 39.076862 39.258415 42.166327 10
+#> B 19.589084 19.631059 19.8682396 19.781567 19.955611 20.751941 10
+#> C 0.255829 0.382732 0.4199913 0.400156 0.499156 0.564807 10
+So going from mo_fullname("Staphylococcus aureus") to "Staphylococcus aureus" takes 0.0004 seconds - it doesn’t even start calculating if the result would be the same as the expected resulting value. That goes for all helper functions:
+microbenchmark(A = mo_species("aureus"),
+ B = mo_genus("Staphylococcus"),
+ C = mo_fullname("Staphylococcus aureus"),
+ D = mo_family("Staphylococcaceae"),
+ E = mo_order("Bacillales"),
+ F = mo_class("Bacilli"),
+ G = mo_phylum("Firmicutes"),
+ H = mo_kingdom("Bacteria"),
+ times = 10,
+ unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean median uq max neval
+#> A 0.250242 0.292496 0.3891774 0.4266960 0.456902 0.520388 10
+#> B 0.259461 0.311702 0.3428727 0.3412800 0.374141 0.443912 10
+#> C 0.290960 0.313169 0.4334429 0.4097595 0.520389 0.725373 10
+#> D 0.271823 0.282789 0.3187217 0.3192800 0.352909 0.375398 10
+#> E 0.245353 0.270985 0.3081197 0.2960235 0.330839 0.429036 10
+#> F 0.246122 0.266585 0.2991101 0.3089435 0.332794 0.351582 10
+#> G 0.271893 0.272452 0.3085039 0.2850580 0.368204 0.385525 10
+#> H 0.252686 0.259251 0.3161791 0.2985025 0.334820 0.422680 10
+Of course, when running mo_phylum("Firmicutes") the function has zero knowledge about the actual microorganism, namely S. aureus. But since the result would be "Firmicutes" too, there is no point in calculating the result. And because this package ‘knows’ all phyla of all known bacteria (according to the Catalogue of Life), it can just return the initial value immediately.
Results in other languages
When the system language is non-English and supported by this AMR package, some functions take a little while longer:
-mo_fullname("CoNS", language = "en") # or just mo_fullname("CoNS") on an English system
-# "Coagulase Negative Staphylococcus (CoNS)"
-
-mo_fullname("CoNS", language = "fr") # or just mo_fullname("CoNS") on a French system
-# "Staphylococcus à coagulase négative (CoNS)"
-
-microbenchmark(en = mo_fullname("CoNS", language = "en"),
- de = mo_fullname("CoNS", language = "de"),
- nl = mo_fullname("CoNS", language = "nl"),
- es = mo_fullname("CoNS", language = "es"),
- it = mo_fullname("CoNS", language = "it"),
- fr = mo_fullname("CoNS", language = "fr"),
- pt = mo_fullname("CoNS", language = "pt"),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# en 6.093583 6.51724 6.555105 6.562986 6.630663 6.99698 100
-# de 13.934874 14.35137 16.891587 14.462210 14.764658 43.63956 100
-# nl 13.900092 14.34729 15.943268 14.424565 14.581535 43.76283 100
-# es 13.833813 14.34596 14.574783 14.439757 14.653994 17.49168 100
-# it 13.811883 14.36621 15.179060 14.453515 14.812359 43.64284 100
-# fr 13.798683 14.37019 16.344731 14.468775 14.697610 48.62923 100
-# pt 13.789674 14.36244 15.706321 14.443772 14.679905 44.76701 100
+mo_fullname("CoNS", language = "en") # or just mo_fullname("CoNS") on an English system
+#> [1] "Coagulase Negative Staphylococcus (CoNS)"
+
+mo_fullname("CoNS", language = "fr") # or just mo_fullname("CoNS") on a French system
+#> [1] "Staphylococcus à coagulase négative (CoNS)"
+
+microbenchmark(en = mo_fullname("CoNS", language = "en"),
+ de = mo_fullname("CoNS", language = "de"),
+ nl = mo_fullname("CoNS", language = "nl"),
+ es = mo_fullname("CoNS", language = "es"),
+ it = mo_fullname("CoNS", language = "it"),
+ fr = mo_fullname("CoNS", language = "fr"),
+ pt = mo_fullname("CoNS", language = "pt"),
+ times = 10,
+ unit = "ms")
+#> Unit: milliseconds
+#> expr min lq mean median uq max neval
+#> en 10.67105 11.03136 11.06332 11.07271 11.15310 11.45006 10
+#> de 19.13393 19.50080 26.13799 19.61419 20.23400 52.66501 10
+#> nl 19.05410 19.53789 22.94707 19.59205 20.12616 52.47399 10
+#> es 19.31635 19.55221 26.22342 19.58633 20.01875 52.97636 10
+#> it 19.21725 19.47105 19.63980 19.58053 19.68162 20.58914 10
+#> fr 19.07854 19.45450 19.67303 19.56153 19.64517 20.45651 10
+#> pt 19.00668 19.28388 19.53493 19.57857 19.66423 20.55317 10
Currently supported are German, Dutch, Spanish, Italian, French and Portuguese.
+
-This package contains the **complete microbial taxonomic data** (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, https://www.itis.gov).
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life ([www.catalogueoflife.org](http://www.catalogueoflife.org)). This data is updated annually - check the included version with `catalogue_of_life_version()`.
-All ~20,000 (sub)species from **the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package**, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
-Read more about the data from ITIS [in our manual](./reference/ITIS.html).
+* All ~55,000 species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses
+* All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera *Aspergillus*, *Candida*, *Pneumocystis*, *Saccharomyces* and *Trichophyton*).
+* All ~15,000 previously accepted names of species that have been taxonomically renamed
+* The responsible author(s) and year of scientific publication
+
+The Catalogue of Life ([www.catalogueoflife.org](http://www.catalogueoflife.org)) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+Read more about the data from the Catalogue of Life [in our manual](./reference/catalogue_of_life.html).
#### Overview of functions
diff --git a/man/ITIS.Rd b/man/ITIS.Rd
deleted file mode 100644
index c9b9bcb3..00000000
--- a/man/ITIS.Rd
+++ /dev/null
@@ -1,53 +0,0 @@
-% Generated by roxygen2: do not edit by hand
-% Please edit documentation in R/itis.R
-\name{ITIS}
-\alias{ITIS}
-\title{ITIS: Integrated Taxonomic Information System}
-\description{
-All taxonomic names of all microorganisms are included in this package, using the authoritative Integrated Taxonomic Information System (ITIS).
-}
-\section{ITIS}{
-
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
-
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
-
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
-}
-
-\section{Read more on our website!}{
-
-\if{html}{\figure{logo.png}{options: height=40px style=margin-bottom:5px} \cr}
-On our website \url{https://msberends.gitlab.io/AMR} you can find \href{https://msberends.gitlab.io/AMR/articles/AMR.html}{a comprehensive tutorial} about how to conduct AMR analysis, the \href{https://msberends.gitlab.io/AMR/reference}{complete documentation of all functions} (which reads a lot easier than here in R) and \href{https://msberends.gitlab.io/AMR/articles/WHONET.html}{an example analysis using WHONET data}.
-}
-
-\examples{
-# Get a note when a species was renamed
-mo_shortname("Chlamydia psittaci")
-# Note: 'Chlamydia psittaci' (Page, 1968) was renamed
-# 'Chlamydophila psittaci' (Everett et al., 1999)
-# [1] "C. psittaci"
-
-# Get any property from the entire taxonomic tree for all included species
-mo_class("E. coli")
-# [1] "Gammaproteobacteria"
-
-mo_family("E. coli")
-# [1] "Enterobacteriaceae"
-
-mo_subkingdom("E. coli")
-# [1] "Negibacteria"
-
-mo_gramstain("E. coli") # based on subkingdom
-# [1] "Gram negative"
-
-mo_ref("E. coli")
-# [1] "Castellani and Chalmers, 1919"
-
-# Do not get mistaken - the package only includes microorganisms
-mo_phylum("C. elegans")
-# [1] "Cyanobacteria" # Bacteria?!
-mo_fullname("C. elegans")
-# [1] "Chroococcus limneticus elegans" # Because a microorganism was found
-}
diff --git a/man/as.mo.Rd b/man/as.mo.Rd
index 662f6996..e62e1150 100644
--- a/man/as.mo.Rd
+++ b/man/as.mo.Rd
@@ -84,7 +84,7 @@ When using \code{allow_uncertain = TRUE} (which is the default setting), it will
\item{It strips off values between brackets and the brackets itself, and re-evaluates the input with all previous rules}
\item{It strips off words from the end one by one and re-evaluates the input with all previous rules}
\item{It strips off words from the start one by one and re-evaluates the input with all previous rules}
- \item{It tries to look for some manual changes which are not yet published to the ITIS database (like \emph{Propionibacterium} not yet being \emph{Cutibacterium})}
+ \item{It tries to look for some manual changes which are not yet published to the Catalogue of Life (like \emph{Propionibacterium} not yet being \emph{Cutibacterium})}
}
Examples:
@@ -107,17 +107,26 @@ Use \code{mo_renamed()} to get a vector with all values that could be coerced ba
[2] Lancefield RC \strong{A serological differentiation of human and other groups of hemolytic streptococci}. 1933. J Exp Med. 57(4): 571–95. \url{https://dx.doi.org/10.1084/jem.57.4.571}
-[3] Integrated Taxonomic Information System (ITIS). Retrieved September 2018. \url{http://www.itis.gov}
+[3] Catalogue of Life: Annual Checklist (public online database), \url{www.catalogueoflife.org}.
}
-\section{ITIS}{
+\section{Catalogue of Life}{
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
}
\section{Read more on our website!}{
@@ -176,7 +185,7 @@ df <- df \%>\%
}
}
\seealso{
-\code{\link{microorganisms}} for the \code{data.frame} with ITIS content that is being used to determine ID's. \cr
+\code{\link{microorganisms}} for the \code{data.frame} that is being used to determine ID's. \cr
The \code{\link{mo_property}} functions (like \code{\link{mo_genus}}, \code{\link{mo_gramstain}}) to get properties based on the returned code.
}
\keyword{Becker}
diff --git a/man/catalogue_of_life.Rd b/man/catalogue_of_life.Rd
new file mode 100644
index 00000000..cb18c5a2
--- /dev/null
+++ b/man/catalogue_of_life.Rd
@@ -0,0 +1,63 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/catalogue_of_life.R
+\name{catalogue_of_life}
+\alias{catalogue_of_life}
+\title{The Catalogue of Life}
+\description{
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life.
+}
+\section{Catalogue of Life}{
+
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
+
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
+
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
+}
+
+\section{Read more on our website!}{
+
+\if{html}{\figure{logo.png}{options: height=40px style=margin-bottom:5px} \cr}
+On our website \url{https://msberends.gitlab.io/AMR} you can find \href{https://msberends.gitlab.io/AMR/articles/AMR.html}{a comprehensive tutorial} about how to conduct AMR analysis, the \href{https://msberends.gitlab.io/AMR/reference}{complete documentation of all functions} (which reads a lot easier than here in R) and \href{https://msberends.gitlab.io/AMR/articles/WHONET.html}{an example analysis using WHONET data}.
+}
+
+\examples{
+# Get version info of included data set
+catalogue_of_life_version()
+
+
+# Get a note when a species was renamed
+mo_shortname("Chlamydia psittaci")
+# Note: 'Chlamydia psittaci' (Page, 1968) was renamed
+# 'Chlamydophila psittaci' (Everett et al., 1999)
+# [1] "C. psittaci"
+
+# Get any property from the entire taxonomic tree for all included species
+mo_class("E. coli")
+# [1] "Gammaproteobacteria"
+
+mo_family("E. coli")
+# [1] "Enterobacteriaceae"
+
+mo_gramstain("E. coli") # based on kingdom and phylum, see ?mo_gramstain
+# [1] "Gram negative"
+
+mo_ref("E. coli")
+# [1] "Castellani et al., 1919"
+
+# Do not get mistaken - the package only includes microorganisms
+mo_phylum("C. elegans")
+# [1] "Cyanobacteria" # Bacteria?!
+mo_fullname("C. elegans")
+# [1] "Chroococcus limneticus elegans" # Because a microorganism was found
+}
diff --git a/man/catalogue_of_life_version.Rd b/man/catalogue_of_life_version.Rd
new file mode 100644
index 00000000..696fa1b6
--- /dev/null
+++ b/man/catalogue_of_life_version.Rd
@@ -0,0 +1,33 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/data.R
+\name{catalogue_of_life_version}
+\alias{catalogue_of_life_version}
+\title{Version info of included Catalogue of Life}
+\usage{
+catalogue_of_life_version()
+}
+\description{
+Version info of included Catalogue of Life
+}
+\section{Catalogue of Life}{
+
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
+
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
+
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
+}
+
+\seealso{
+\code{\link{microorganisms}}
+}
diff --git a/man/figures/logo_col.png b/man/figures/logo_col.png
new file mode 100644
index 00000000..b6a27197
Binary files /dev/null and b/man/figures/logo_col.png differ
diff --git a/man/figures/logo_itis.jpg b/man/figures/logo_itis.jpg
deleted file mode 100644
index e2788e42..00000000
Binary files a/man/figures/logo_itis.jpg and /dev/null differ
diff --git a/man/figures/plus.png b/man/figures/plus.png
deleted file mode 100644
index 357f17a4..00000000
Binary files a/man/figures/plus.png and /dev/null differ
diff --git a/man/microorganisms.Rd b/man/microorganisms.Rd
index c8731931..1a1a2b27 100755
--- a/man/microorganisms.Rd
+++ b/man/microorganisms.Rd
@@ -4,26 +4,25 @@
\name{microorganisms}
\alias{microorganisms}
\title{Data set with ~60,000 microorganisms}
-\format{A \code{\link{data.frame}} with 56,659 observations and 15 variables:
+\format{A \code{\link{data.frame}} with 56,672 observations and 14 variables:
\describe{
- \item{\code{mo}}{ID of microorganism}
+ \item{\code{mo}}{ID of microorganism as used by this package}
\item{\code{col_id}}{Catalogue of Life ID}
- \item{\code{genus}}{Taxonomic genus of the microorganism as found in ITIS, see Source}
- \item{\code{species}}{Taxonomic species of the microorganism as found in ITIS, see Source}
- \item{\code{subspecies}}{Taxonomic subspecies of the microorganism as found in ITIS, see Source}
\item{\code{fullname}}{Full name, like \code{"Echerichia coli"}}
- \item{\code{family}}{Taxonomic family of the microorganism as found in ITIS, see Source}
- \item{\code{order}}{Taxonomic order of the microorganism as found in ITIS, see Source}
- \item{\code{class}}{Taxonomic class of the microorganism as found in ITIS, see Source}
- \item{\code{phylum}}{Taxonomic phylum of the microorganism as found in ITIS, see Source}
- \item{\code{subkingdom}}{Taxonomic subkingdom of the microorganism as found in ITIS, see Source}
- \item{\code{kingdom}}{Taxonomic kingdom of the microorganism as found in ITIS, see Source}
- \item{\code{gramstain}}{Gram of microorganism, like \code{"Gram negative"}}
- \item{\code{prevalence}}{An integer based on estimated prevalence of the microorganism in humans. Used internally by \code{\link{as.mo}}, otherwise quite meaningless. It has a value of 25 for manually added items and a value of 1000 for all unprevalent microorganisms whose genus was somewhere in the top 250 (with another species).}
- \item{\code{ref}}{Author(s) and year of concerning publication as found in ITIS, see Source}
+ \item{\code{kingdom}}{Taxonomic kingdom of the microorganism}
+ \item{\code{phylum}}{Taxonomic phylum of the microorganism}
+ \item{\code{class}}{Taxonomic class of the microorganism}
+ \item{\code{order}}{Taxonomic order of the microorganism}
+ \item{\code{family}}{Taxonomic family of the microorganism}
+ \item{\code{genus}}{Taxonomic genus of the microorganism}
+ \item{\code{species}}{Taxonomic species of the microorganism}
+ \item{\code{subspecies}}{Taxonomic subspecies of the microorganism}
+ \item{\code{rank}}{Taxonomic rank of the microorganism, like \code{"species"} or \code{"genus"}}
+ \item{\code{ref}}{Author(s) and year of concerning scientific publication}
+ \item{\code{species_id}}{ID of the species as used by the Catalogue of Life}
}}
\source{
-Integrated Taxonomic Information System (ITIS) public online database, \url{https://www.itis.gov}.
+Catalogue of Life: Annual Checklist (public online database), \url{www.catalogueoflife.org}.
}
\usage{
microorganisms
@@ -39,14 +38,23 @@ Manually added were:
\item{2 other undefined (unknown Gram negatives and unknown Gram positives)}
}
}
-\section{ITIS}{
+\section{Catalogue of Life}{
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
}
\section{Read more on our website!}{
@@ -56,6 +64,6 @@ On our website \url{https://msberends.gitlab.io/AMR} you can find \href{https://
}
\seealso{
-\code{\link{as.mo}} \code{\link{mo_property}} \code{\link{microorganisms.codes}}
+\code{\link{as.mo}}, \code{\link{mo_property}}, \code{\link{microorganisms.codes}}
}
\keyword{datasets}
diff --git a/man/microorganisms.codes.Rd b/man/microorganisms.codes.Rd
index a7e1a2a9..7357f3d1 100644
--- a/man/microorganisms.codes.Rd
+++ b/man/microorganisms.codes.Rd
@@ -15,14 +15,23 @@ microorganisms.codes
\description{
A data set containing commonly used codes for microorganisms, from laboratory systems and WHONET. Define your own with \code{\link{set_mo_source}}.
}
-\section{ITIS}{
+\section{Catalogue of Life}{
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
}
\section{Read more on our website!}{
diff --git a/man/microorganisms.old.Rd b/man/microorganisms.old.Rd
index 315220c4..49dba11c 100644
--- a/man/microorganisms.old.Rd
+++ b/man/microorganisms.old.Rd
@@ -8,26 +8,35 @@
\describe{
\item{\code{col_id}}{Catalogue of Life ID}
\item{\code{tsn_new}}{New Catalogue of Life ID}
- \item{\code{fullname}}{Old taxonomic name of the microorganism as found in the CoL, see Source}
- \item{\code{ref}}{Author(s) and year of concerning publication as found in the CoL, see Source}
+ \item{\code{fullname}}{Old taxonomic name of the microorganism}
+ \item{\code{ref}}{Author(s) and year of concerning scientific publication}
}}
\source{
-[3] Integrated Taxonomic Information System (ITIS) on-line database, \url{https://www.itis.gov}.
+[3] Catalogue of Life: Annual Checklist (public online database), \url{www.catalogueoflife.org}.
}
\usage{
microorganisms.old
}
\description{
-A data set containing old (previously valid or accepted) taxonomic names according to ITIS. This data set is used internally by \code{\link{as.mo}}.
+A data set containing old (previously valid or accepted) taxonomic names according to the Catalogue of Life. This data set is used internally by \code{\link{as.mo}}.
}
-\section{ITIS}{
+\section{Catalogue of Life}{
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
}
\section{Read more on our website!}{
diff --git a/man/mo_property.Rd b/man/mo_property.Rd
index ab81734a..d7509804 100644
--- a/man/mo_property.Rd
+++ b/man/mo_property.Rd
@@ -11,7 +11,6 @@
\alias{mo_order}
\alias{mo_class}
\alias{mo_phylum}
-\alias{mo_subkingdom}
\alias{mo_kingdom}
\alias{mo_type}
\alias{mo_gramstain}
@@ -19,6 +18,7 @@
\alias{mo_authors}
\alias{mo_year}
\alias{mo_taxonomy}
+\alias{mo_url}
\title{Property of a microorganism}
\usage{
mo_fullname(x, language = get_locale(), ...)
@@ -39,8 +39,6 @@ mo_class(x, ...)
mo_phylum(x, ...)
-mo_subkingdom(x, ...)
-
mo_kingdom(x, ...)
mo_type(x, language = get_locale(), ...)
@@ -55,6 +53,8 @@ mo_year(x, ...)
mo_taxonomy(x, ...)
+mo_url(x, ...)
+
mo_property(x, property = "fullname", language = get_locale(), ...)
}
\arguments{
@@ -77,26 +77,39 @@ mo_property(x, property = "fullname", language = get_locale(), ...)
Use these functions to return a specific property of a microorganism from the \code{\link{microorganisms}} data set. All input values will be evaluated internally with \code{\link{as.mo}}.
}
\details{
-All functions will return the most recently known taxonomic property according to ITIS, except for \code{mo_ref}, \code{mo_authors} and \code{mo_year}. This leads to the following results:
+All functions will return the most recently known taxonomic property according to the Catalogue of Life, except for \code{mo_ref}, \code{mo_authors} and \code{mo_year}. This leads to the following results:
\itemize{
\item{\code{mo_fullname("Chlamydia psittaci")} will return \code{"Chlamydophila psittaci"} (with a warning about the renaming)}
\item{\code{mo_ref("Chlamydia psittaci")} will return \code{"Page, 1968"} (with a warning about the renaming)}
\item{\code{mo_ref("Chlamydophila psittaci")} will return \code{"Everett et al., 1999"} (without a warning)}
}
+
+The Gram stain - \code{mo_gramstain()} - will be determined on the taxonomic kingdom and phylum. According to Cavalier-Smith (2002) who defined subkingdoms Negibacteria and Posibacteria, only these phyla are Posibacteria: Actinobacteria, Chloroflexi, Firmicutes and Tenericutes (ref: \url{https://itis.gov/servlet/SingleRpt/SingleRpt?search_topic=TSN&search_value=956097}). These bacteria are considered Gram positive - all other bacteria are considered Gram negative. Species outside the kingdom of Bacteria will return a value \code{NA}.
+
+The function \code{mo_url()} will return the direct URL to the species in the Catalogue of Life.
}
\section{Supported languages}{
Supported languages are \code{"en"} (English), \code{"de"} (German), \code{"nl"} (Dutch), \code{"es"} (Spanish), \code{"it"} (Italian), \code{"fr"} (French), and \code{"pt"} (Portuguese).
}
-\section{ITIS}{
+\section{Catalogue of Life}{
-\if{html}{\figure{logo_itis.jpg}{options: height=60px style=margin-bottom:5px} \cr}
-This package contains the \strong{complete microbial taxonomic data} (with all nine taxonomic ranks - from kingdom to subspecies) from the publicly available Integrated Taxonomic Information System (ITIS, \url{https://www.itis.gov}).
+\if{html}{\figure{logo_col.png}{options: height=60px style=margin-bottom:5px} \cr}
+This package contains the complete taxonomic tree of almost all microorganisms from the authoritative and comprehensive Catalogue of Life (\url{http://www.catalogueoflife.org}). This data is updated annually - check the included version with \code{\link{catalogue_of_life_version}}.
-All ~20,000 (sub)species from \strong{the taxonomic kingdoms Bacteria, Fungi and Protozoa are included in this package}, as well as all their ~2,500 previously accepted names known to ITIS. Furthermore, the responsible authors and year of publication are available. This allows users to use authoritative taxonomic information for their data analysis on any microorganism, not only human pathogens. It also helps to quickly determine the Gram stain of bacteria, since ITIS honours the taxonomic branching order of bacterial phyla according to Cavalier-Smith (2002), which defines that all bacteria are classified into either subkingdom Negibacteria or subkingdom Posibacteria.
+Included are:
+\itemize{
+ \item{All ~55,000 (sub)species from the kingdoms of Archaea, Bacteria, Protozoa and Viruses}
+ \item{All ~3,000 (sub)species from these orders of the kingdom of Fungi: Eurotiales, Onygenales, Pneumocystales, Saccharomycetales and Schizosaccharomycetales. The kingdom of Fungi is a very large taxon with almost 300,000 different species, of which most are not microbial. Including everything tremendously slows down our algortihms, and not all fungi fit the scope of this package. By only including the aforementioned taxonomic orders, the most relevant species are covered (like genera \emph{Aspergillus}, \emph{Candida}, \emph{Pneumocystis}, \emph{Saccharomyces} and \emph{Trichophyton}).}
+ \item{All ~15,000 previously accepted names of (sub)species that have been taxonomically renamed}
+ \item{The complete taxonomic tree of all included (sub)species: from kingdom to subspecies}
+ \item{The responsible author(s) and year of scientific publication}
+}
-ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic specialists [3].
+The Catalogue of Life (\url{http://www.catalogueoflife.org}) is the most comprehensive and authoritative global index of species currently available. It holds essential information on the names, relationships and distributions of over 1.6 million species. The Catalogue of Life is used to support the major biodiversity and conservation information services such as the Global Biodiversity Information Facility (GBIF), Encyclopedia of Life (EoL) and the International Union for Conservation of Nature Red List. It is recognised by the Convention on Biological Diversity as a significant component of the Global Taxonomy Initiative and a contribution to Target 1 of the Global Strategy for Plant Conservation.
+
+The syntax used to transform the original data to a cleansed R format, can be found here: \url{https://gitlab.com/msberends/AMR/blob/master/reproduction_of_microorganisms.R}.
}
\section{Source}{
@@ -105,7 +118,7 @@ ITIS is a partnership of U.S., Canadian, and Mexican agencies and taxonomic spec
[2] Lancefield RC \strong{A serological differentiation of human and other groups of hemolytic streptococci}. 1933. J Exp Med. 57(4): 571–95. \url{https://dx.doi.org/10.1084/jem.57.4.571}
-[3] Integrated Taxonomic Information System (ITIS). Retrieved September 2018. \url{http://www.itis.gov}
+[3] Catalogue of Life: Annual Checklist (public online database), \url{www.catalogueoflife.org}.
}
\section{Read more on our website!}{
@@ -118,14 +131,13 @@ On our website \url{https://msberends.gitlab.io/AMR} you can find \href{https://
# All properties of Escherichia coli
## taxonomic properties
mo_kingdom("E. coli") # "Bacteria"
-mo_subkingdom("E. coli") # "Negibacteria"
mo_phylum("E. coli") # "Proteobacteria"
mo_class("E. coli") # "Gammaproteobacteria"
mo_order("E. coli") # "Enterobacteriales"
mo_family("E. coli") # "Enterobacteriaceae"
mo_genus("E. coli") # "Escherichia"
mo_species("E. coli") # "coli"
-mo_subspecies("E. coli") # NA
+mo_subspecies("E. coli") # ""
## colloquial properties
mo_fullname("E. coli") # "Escherichia coli"
diff --git a/reproduction_of_microorganisms.R b/reproduction_of_microorganisms.R
new file mode 100644
index 00000000..462393db
--- /dev/null
+++ b/reproduction_of_microorganisms.R
@@ -0,0 +1,256 @@
+# Catalogue of Life
+# Data retrieved from Encyclopaedia of Life:
+# https://opendata.eol.org/dataset/catalogue-of-life/
+
+# unzip and extract taxon.tab, then:
+taxon <- data.table::fread("taxon.tab")
+# result is over 3.7M rows
+taxon %>% freq(kingdom)
+# Item Count Percent Cum. Count Cum. Percent
+# --- ---------- ---------- -------- ----------- -------------
+# 1 Animalia 2,225,627 59.1% 2,225,627 59.1%
+# 2 Plantae 1,177,412 31.3% 3,403,039 90.4%
+# 3 Fungi 290,145 7.7% 3,693,184 98.1%
+# 4 Chromista 47,126 1.3% 3,740,310 99.3%
+# 5 Bacteria 14,478 0.4% 3,754,788 99.7%
+# 6 Protozoa 6,060 0.2% 3,760,848 99.9%
+# 7 Viruses 3,827 0.1% 3,764,675 100.0%
+# 8 Archaea 610 0.0% 3,765,285 100.0%
+
+MOs <- taxon %>%
+ # tibble for future transformations
+ as_tibble() %>%
+ filter(
+ # we only want all microorganisms and viruses
+ !kingdom %in% c("Animalia", "Plantae"),
+ # and no entries above genus - they all already have a taxonomic tree
+ !taxonRank %in% c("kingdom", "phylum", "superfamily", "class", "order", "family"),
+ # not all fungi: Aspergillus, Candida, Trichphyton and Pneumocystis are the most important,
+ # so only keep these orders from the fungi:
+ !(kingdom == "Fungi" & !order %in% c("Eurotiales", "Saccharomycetales", "Schizosaccharomycetales", "Onygenales", "Pneumocystales"))) %>%
+ # remove text if it contains 'Not assigned' like phylum in viruses
+ mutate_all(funs(gsub("Not assigned", "", .))) %>%
+ # only latest ref, not original authors
+ mutate(scientificNameAuthorship = trimws(gsub(".*[)] ", "", scientificNameAuthorship)),
+ scientificNameAuthorship = ifelse(grepl(" emend[. ]", scientificNameAuthorship, ignore.case = TRUE),
+ gsub("(.*)emend[. ]+(.*)", "\\2", scientificNameAuthorship, ignore.case = TRUE),
+ scientificNameAuthorship),
+ scientificNameAuthorship = gsub(".", "", scientificNameAuthorship, fixed = TRUE),
+ scientificNameAuthorship = gsub(",? et al", " et al.", scientificNameAuthorship, fixed = FALSE, ignore.case = TRUE),
+ scientificNameAuthorship = gsub("[()]", "", scientificNameAuthorship),
+ # year always preceded by comma
+ scientificNameAuthorship = gsub(" ([0-9]{4})$", ", \\1", scientificNameAuthorship),
+ scientificNameAuthorship = gsub(",,", ",", scientificNameAuthorship, fixed = TRUE),
+ # only first author with *et al.*
+ scientificNameAuthorship = gsub(",.*,", " et al.,", scientificNameAuthorship),
+ scientificNameAuthorship = gsub(" (and|&) .*,", " et al.,", scientificNameAuthorship),
+ scientificNameAuthorship = gsub(", [^0-9]+", ", ", scientificNameAuthorship),
+ scientificNameAuthorship = gsub(", $", "", scientificNameAuthorship)
+ )
+
+# remove non-ASCII characters (not allowed by CRAN)
+MOs <- MOs %>%
+ lapply(iconv, from = "UTF-8", to = "ASCII//TRANSLIT") %>%
+ as_tibble(stringsAsFactors = FALSE)
+
+# split old taxonomic names - they refer to a new `taxonID` with `acceptedNameUsageID`
+MOs.old <- MOs %>%
+ filter(!is.na(acceptedNameUsageID),
+ scientificNameAuthorship != "") %>%
+ transmute(col_id = taxonID,
+ col_id_new = acceptedNameUsageID,
+ fullname =
+ trimws(
+ gsub("(.*)[(].*", "\\1",
+ stringr::str_replace(
+ string = scientificName,
+ pattern = stringr::fixed(scientificNameAuthorship),
+ replacement = ""))),
+ ref = scientificNameAuthorship) %>%
+ filter(!is.na(fullname)) %>%
+ distinct(fullname, .keep_all = TRUE) %>%
+ arrange(col_id)
+
+MOs <- MOs %>%
+ filter(is.na(acceptedNameUsageID)) %>%
+ transmute(col_id = taxonID,
+ fullname = trimws(ifelse(kingdom == "Viruses",
+ paste(specificEpithet, infraspecificEpithet),
+ paste(genus, specificEpithet, infraspecificEpithet))),
+ kingdom,
+ phylum,
+ class,
+ order,
+ family,
+ genus = gsub(":", "", genus),
+ species = specificEpithet,
+ subspecies = infraspecificEpithet,
+ rank = taxonRank,
+ ref = scientificNameAuthorship,
+ species_id = gsub(".*/([a-f0-9]+)", "\\1", furtherInformationURL)) %>%
+ distinct(fullname, .keep_all = TRUE) %>%
+ filter(!grepl("unassigned", fullname, ignore.case = TRUE))
+
+# only old names of species that are in MOs:
+MOs.old <- MOs.old %>% filter(col_id_new %in% MOs$col_id)
+
+MOs <- MOs %>%
+ group_by(kingdom) %>%
+ # abbreviations may be same for genera between kingdoms,
+ # because each abbreviation starts with the the first character of the kingdom
+ mutate(abbr_genus = abbreviate(genus,
+ minlength = 5,
+ use.classes = TRUE,
+ method = "both.sides",
+ strict = FALSE)) %>%
+ ungroup() %>%
+ group_by(genus) %>%
+ # species abbreviations may be the same between genera
+ # because the genus abbreviation is part of the abbreviation
+ mutate(abbr_species = abbreviate(species,
+ minlength = 3,
+ use.classes = FALSE,
+ method = "both.sides")) %>%
+ ungroup() %>%
+ group_by(genus, species) %>%
+ mutate(abbr_subspecies = abbreviate(subspecies,
+ minlength = 3,
+ use.classes = FALSE,
+ method = "both.sides")) %>%
+ ungroup() %>%
+ # remove trailing underscores
+ mutate(mo = gsub("_+$", "",
+ toupper(paste(substr(kingdom, 1, 1),
+ abbr_genus,
+ abbr_species,
+ abbr_subspecies,
+ sep = "_")))) %>%
+ mutate(mo = ifelse(duplicated(.$mo), paste0(mo, "1"), mo)) %>%
+ select(mo, everything(), -abbr_genus, -abbr_species, -abbr_subspecies)
+
+# everything distinct?
+sum(duplicated(MOs$mo))
+
+# add non-taxonomic entries
+MOs <- MOs %>%
+ bind_rows(
+ # CoNS
+ MOs %>%
+ filter(genus == "Staphylococcus", species == "epidermidis") %>% .[1,] %>%
+ mutate(mo = gsub("EPI", "CNS", mo),
+ col_id = NA_integer_,
+ species = "coagulase negative",
+ fullname = "Coagulase Negative Staphylococcus (CoNS)",
+ ref = NA_character_),
+ # CoPS
+ MOs %>%
+ filter(genus == "Staphylococcus", species == "epidermidis") %>% .[1,] %>%
+ mutate(mo = gsub("EPI", "CPS", mo),
+ col_id = NA_integer_,
+ species = "coagulase positive",
+ fullname = "Coagulase Positive Staphylococcus (CoPS)",
+ ref = NA_character_),
+ # Streptococci groups A, B, C, F, H, K
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRA", mo),
+ col_id = NA_integer_,
+ species = "group A" ,
+ fullname = "Streptococcus group A"),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "dysgalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("DYS", "GRB", mo),
+ col_id = NA_integer_,
+ species = "group B" ,
+ fullname = "Streptococcus group B"),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRC", mo),
+ col_id = NA_integer_,
+ species = "group C" ,
+ fullname = "Streptococcus group C",
+ ref = NA_character_),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRD", mo),
+ col_id = NA_integer_,
+ species = "group D" ,
+ fullname = "Streptococcus group D",
+ ref = NA_character_),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRF", mo),
+ col_id = NA_integer_,
+ species = "group F" ,
+ fullname = "Streptococcus group F",
+ ref = NA_character_),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRG", mo),
+ col_id = NA_integer_,
+ species = "group F" ,
+ fullname = "Streptococcus group G",
+ ref = NA_character_),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRH", mo),
+ col_id = NA_integer_,
+ species = "group H" ,
+ fullname = "Streptococcus group H",
+ ref = NA_character_),
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "GRK", mo),
+ col_id = NA_integer_,
+ species = "group K" ,
+ fullname = "Streptococcus group K",
+ ref = NA_character_),
+ # Beta haemolytic Streptococci
+ MOs %>%
+ filter(genus == "Streptococcus", species == "agalactiae") %>% .[1,] %>%
+ mutate(mo = gsub("AGA", "HAE", mo),
+ col_id = NA_integer_,
+ species = "beta-haemolytic" ,
+ fullname = "Beta-haemolytic Streptococcus",
+ ref = NA_character_),
+ # unknowns
+ data.frame(mo = "B_GRAMN",
+ col_id = NA_integer_,
+ fullname = "(unknown Gram negatives)",
+ kingdom = "Bacteria",
+ phylum = NA_character_,
+ class = NA_character_,
+ order = NA_character_,
+ family = NA_character_,
+ genus = "(unknown Gram negatives)",
+ species = NA_character_,
+ subspecies = NA_character_,
+ rank = "species",
+ ref = NA_character_,
+ stringsAsFactors = FALSE),
+ data.frame(mo = "B_GRAMP",
+ col_id = NA_integer_,
+ fullname = "(unknown Gram positives)",
+ kingdom = "Bacteria",
+ phylum = NA_character_,
+ class = NA_character_,
+ order = NA_character_,
+ family = NA_character_,
+ genus = "(unknown Gram positives)",
+ species = NA_character_,
+ subspecies = NA_character_,
+ rank = "species",
+ ref = NA_character_,
+ stringsAsFactors = FALSE)
+ )
+
+# save it
+MOs <- as.data.frame(MOs %>% arrange(mo), stringsAsFactors = FALSE)
+MOs.old <- as.data.frame(MOs.old, stringsAsFactors = FALSE)
+class(MOs$mo) <- "mo"
+
+saveRDS(MOs, "microorganisms.rds")
+saveRDS(MOs.old, "microorganisms.old.rds")
+# on the server:
+# usethis::use_data(microorganisms, overwrite = TRUE)
+# usethis::use_data(microorganisms.old, overwrite = TRUE)
diff --git a/vignettes/benchmarks.Rmd b/vignettes/benchmarks.Rmd
index 3e8b1001..6a5c0ee3 100755
--- a/vignettes/benchmarks.Rmd
+++ b/vignettes/benchmarks.Rmd
@@ -1,7 +1,7 @@
---
title: "Benchmarks"
author: "Matthijs S. Berends"
-date: "14 February 2019"
+date: '`r format(Sys.Date(), "%d %B %Y")`'
output:
rmarkdown::html_vignette:
toc: true
@@ -17,17 +17,17 @@ editor_options:
```{r setup, include = FALSE, results = 'markup'}
knitr::opts_chunk$set(
collapse = TRUE,
- comment = "#",
+ comment = "#>",
fig.width = 7.5,
fig.height = 4.5
)
```
-One of the most important features of this package is the complete microbial taxonomic database, supplied by ITIS (https://www.itis.gov). We created a function `as.mo()` that transforms any user input value to a valid microbial ID by using AI (Artificial Intelligence) and based on the taxonomic tree of ITIS.
+One of the most important features of this package is the complete microbial taxonomic database, supplied by the Catalogue of Life (http://catalogueoflife.org). We created a function `as.mo()` that transforms any user input value to a valid microbial ID by using AI (Artificial Intelligence) combined with the taxonomic tree of Catalogue of Life.
-Using the `microbenchmark` package, we can review the calculation performance of this function. Its function `microbenchmark()` calculates different input expressions independently of each others and runs every expression 100 times.
+Using the `microbenchmark` package, we can review the calculation performance of this function. Its function `microbenchmark()` runs different input expressions independently of each other and measures their time-to-result.
-```{r, eval = FALSE}
+```{r}
library(microbenchmark)
library(AMR)
```
@@ -36,73 +36,57 @@ In the next test, we try to 'coerce' different input values for *Staphylococcus
But the calculation time differs a lot. Here, the AI effect can be reviewed best:
-```{r, eval = FALSE}
-benchmark <- microbenchmark(as.mo("sau"),
- as.mo("stau"),
- as.mo("staaur"),
- as.mo("S. aureus"),
- as.mo("S. aureus"),
- as.mo("STAAUR"),
- as.mo("Staphylococcus aureus"),
- as.mo("B_STPHY_AUR"))
-print(benchmark, unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# as.mo("sau") 18.983141 19.121148 19.9676944 19.1967505 19.2871260 38.635012 100
-# as.mo("stau") 37.503863 37.692049 38.9856547 37.8244335 37.9851040 57.576107 100
-# as.mo("staaur") 18.945427 19.122579 19.6392560 19.2241285 19.3536140 38.687672 100
-# as.mo("S. aureus") 15.305229 15.471103 16.3477096 15.5545630 15.6689280 36.363005 100
-# as.mo("S. aureus") 15.308232 15.469881 16.5269706 15.5506870 15.6277560 42.155292 100
-# as.mo("STAAUR") 18.984049 19.117166 19.6104597 19.2219285 19.3161095 38.638783 100
-# as.mo("Staphylococcus aureus") 8.103546 8.198285 8.6422018 8.2636915 8.3200535 27.002527 100
-# as.mo("B_STPHY_AUR") 0.156236 0.196779 0.2017926 0.2035535 0.2115505 0.241861 100
+```{r}
+S.aureus <- microbenchmark(as.mo("sau"),
+ as.mo("stau"),
+ as.mo("staaur"),
+ as.mo("S. aureus"),
+ as.mo("S. aureus"),
+ as.mo("STAAUR"),
+ as.mo("Staphylococcus aureus"),
+ as.mo("B_STPHY_AUR"),
+ times = 10)
+print(S.aureus, unit = "ms")
+```
-par(mar = c(5, 15, 4, 2)) # set more space for left margin text (15)
-boxplot(benchmark, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, 200),
+In the table above, all measurements are shown in milliseconds (thousands of seconds). A value of 10 milliseconds means it can determine 100 input values per second. It case of 50 milliseconds, this is only 20 input values per second. The more an input value resembles a full name, the faster the result will be found. In case of `as.mo("B_STPHY_AUR")`, the input is already a valid MO code, so it only almost takes no time at all (`r as.integer(min(S.aureus$time, na.rm = TRUE) / 1000)` millionths of seconds).
+
+To achieve this speed, the `as.mo` function also takes into account the prevalence of human pathogenic microorganisms. The downside is of course that less prevalent microorganisms will be determined far less faster. See this example for the ID of *Mycoplasma leonicaptivi* (`B_MYCPL_LEO`), a bug probably never found before in humans:
+
+```{r}
+M.leonicaptivi <- microbenchmark(as.mo("myle"),
+ as.mo("mycleo"),
+ as.mo("M. leonicaptivi"),
+ as.mo("M. leonicaptivi"),
+ as.mo("MYCLEO"),
+ as.mo("Mycoplasma leonicaptivi"),
+ as.mo("B_MYCPL_LEO"),
+ times = 10)
+print(M.leonicaptivi, unit = "ms")
+```
+
+That takes `r round(mean(M.leonicaptivi$time, na.rm = TRUE) / mean(S.aureus$time, na.rm = TRUE), 1)` times as much time on average! A value of 100 milliseconds means it can only determine ~10 different input values per second. We can conclude that looking up arbitrary codes of less prevalent microorganisms is the worst way to go, in terms of calculation performance:
+
+```{r}
+par(mar = c(5, 16, 4, 2)) # set more space for left margin text (16)
+
+# highest value on y axis
+max_y_axis <- max(S.aureus$time, M.leonicaptivi$time, na.rm = TRUE) / 1e6
+
+boxplot(S.aureus, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, max_y_axis),
main = expression(paste("Benchmark of ", italic("Staphylococcus aureus"))))
+
+boxplot(M.leonicaptivi, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, max_y_axis),
+ main = expression(paste("Benchmark of ", italic("Mycoplasma leonicaptivi"))))
```
-
-
-In the table above, all measurements are shown in milliseconds (thousands of seconds), tested on a quite regular Linux server from 2007 (Core 2 Duo 2.7 GHz, 2 GB DDR2 RAM). A value of 8 milliseconds means it can determine 125 input values per second. It case of 40 milliseconds, this is only 25 input values per second. The more an input value resembles a full name, the faster the result will be found. In case of `as.mo("B_STPHY_AUR")`, the input is already a valid MO code, so it only almost takes no time at all (0.0002 seconds on our server).
-
-To achieve this speed, the `as.mo` function also takes into account the prevalence of human pathogenic microorganisms. The downside is of course that less prevalent microorganisms will be determined far less faster. See this example for the ID of *Burkholderia nodosa* (`B_BRKHL_NOD`):
-
-```{r, eval = FALSE}
-benchmark <- microbenchmark(as.mo("buno"),
- as.mo("burnod"),
- as.mo("B. nodosa"),
- as.mo("B. nodosa"),
- as.mo("BURNOD"),
- as.mo("Burkholderia nodosa"),
- as.mo("B_BRKHL_NOD"))
-print(benchmark, unit = "ms")
-
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# as.mo("buno") 125.141333 125.8553210 129.5727691 126.3899910 127.0954925 194.51985 100
-# as.mo("burnod") 142.300359 144.1611750 147.0642288 144.6074960 145.5243025 176.91649 100
-# as.mo("B. nodosa") 81.530132 81.9360840 83.3915418 82.1852770 82.6848870 102.63184 100
-# as.mo("B. nodosa") 81.109547 81.9836805 84.7595894 82.3437825 82.8282705 110.67036 100
-# as.mo("BURNOD") 143.163527 143.9134485 148.7192688 144.5582580 145.7489115 314.92070 100
-# as.mo("Burkholderia nodosa") 36.226325 36.5499000 37.1309929 36.6581540 36.7551985 56.25597 100
-# as.mo("B_BRKHL_NOD") 0.172509 0.3038455 0.4806591 0.3078265 0.3121215 19.16173 100
-
-boxplot(benchmark, horizontal = TRUE, las = 1, unit = "ms", log = FALSE, xlab = "", ylim = c(0, 200),
- main = expression(paste("Benchmark of ", italic("Burkholderia nodosa"))))
-```
-
-
-
-That takes up to 8 times as much time! A value of 145 milliseconds means it can only determine ~7 different input values per second. We can conclude that looking up arbitrary codes of less prevalent microorganisms is the worst way to go, in terms of calculation performance.
-
To relieve this pitfall and further improve performance, two important calculations take almost no time at all: **repetitive results** and **already precalculated results**.
### Repetitive results
-Repetitive results mean that unique values are present more than once. Unique values will only be calculated once by `as.mo()`. We will use `mo_fullname()` for this test - a helper function that returns the full microbial name (genus, species and possibly subspecies) and uses `as.mo()` internally.
+Repetitive results mean that unique values are present more than once. Unique values will only be calculated once by `as.mo()`. We will use `mo_fullname()` for this test - a helper function that returns the full microbial name (genus, species and possibly subspecies) which uses `as.mo()` internally.
-```{r, eval = FALSE}
+```{r}
library(dplyr)
# take 500,000 random MO codes from the septic_patients data set
x = septic_patients %>%
@@ -111,53 +95,33 @@ x = septic_patients %>%
# got the right length?
length(x)
-# [1] 500000
# and how many unique values do we have?
n_distinct(x)
-# [1] 96
-# only 96, but distributed in 500,000 results. now let's see:
-microbenchmark(X = mo_fullname(x),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# X 114.9342 117.1076 129.6448 120.2047 131.5005 168.6371 10
+# now let's see:
+run_it <- microbenchmark(X = mo_fullname(x),
+ times = 10)
+print(run_it, unit = "ms")
```
-So transforming 500,000 values (!) of 96 unique values only takes 0.12 seconds (120 ms). You only lose time on your unique input values.
-
-Results of a tenfold - 5,000,000 values:
-
-```{r, eval = FALSE}
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# X 882.9045 901.3011 1001.677 940.3421 1168.088 1226.846 10
-```
-
-Even determining the full names of 5 *Million* values is done within a second.
+So transforming 500,000 values (!) of `r n_distinct(x)` unique values only takes `r round(median(run_it$time, na.rm = TRUE) / 1e9, 2)` seconds (`r as.integer(median(run_it$time, na.rm = TRUE) / 1e6)` ms). You only lose time on your unique input values.
### Precalculated results
What about precalculated results? If the input is an already precalculated result of a helper function like `mo_fullname()`, it almost doesn't take any time at all (see 'C' below):
-```{r, eval = FALSE}
-microbenchmark(A = mo_fullname("B_STPHY_AUR"),
- B = mo_fullname("S. aureus"),
- C = mo_fullname("Staphylococcus aureus"),
- times = 10,
- unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# A 11.364086 11.460537 11.5104799 11.4795330 11.524860 11.818263 10
-# B 11.976454 12.012352 12.1704592 12.0853020 12.210004 12.881737 10
-# C 0.095823 0.102528 0.1167754 0.1153785 0.132629 0.140661 10
+```{r}
+run_it <- microbenchmark(A = mo_fullname("B_STPHY_AUR"),
+ B = mo_fullname("S. aureus"),
+ C = mo_fullname("Staphylococcus aureus"),
+ times = 10)
+print(run_it, unit = "ms")
```
-So going from `mo_fullname("Staphylococcus aureus")` to `"Staphylococcus aureus"` takes 0.0001 seconds - it doesn't even start calculating *if the result would be the same as the expected resulting value*. That goes for all helper functions:
+So going from `mo_fullname("Staphylococcus aureus")` to `"Staphylococcus aureus"` takes `r format(round(run_it %>% filter(expr == "C") %>% pull(time) %>% median() / 1e9, 4), scientific = FALSE)` seconds - it doesn't even start calculating *if the result would be the same as the expected resulting value*. That goes for all helper functions:
-```{r, eval = FALSE}
+```{r}
microbenchmark(A = mo_species("aureus"),
B = mo_genus("Staphylococcus"),
C = mo_fullname("Staphylococcus aureus"),
@@ -165,35 +129,21 @@ microbenchmark(A = mo_species("aureus"),
E = mo_order("Bacillales"),
F = mo_class("Bacilli"),
G = mo_phylum("Firmicutes"),
- H = mo_subkingdom("Posibacteria"),
- I = mo_kingdom("Bacteria"),
+ H = mo_kingdom("Bacteria"),
times = 10,
unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# A 0.105181 0.121314 0.1478538 0.1465265 0.166711 0.211409 10
-# B 0.132558 0.146388 0.1584278 0.1499835 0.164895 0.208477 10
-# C 0.135492 0.160355 0.2341847 0.1884665 0.348857 0.395931 10
-# D 0.109650 0.115727 0.1270481 0.1264130 0.128648 0.168317 10
-# E 0.081574 0.096940 0.0992582 0.0980915 0.101479 0.120477 10
-# F 0.081575 0.088489 0.0988463 0.0989650 0.103365 0.126482 10
-# G 0.091981 0.095333 0.1043568 0.1001530 0.111327 0.129625 10
-# H 0.092610 0.093169 0.1009135 0.0985455 0.101828 0.120406 10
-# I 0.087371 0.091213 0.1069758 0.0941815 0.109302 0.192831 10
```
-Of course, when running `mo_phylum("Firmicutes")` the function has zero knowledge about the actual microorganism, namely *S. aureus*. But since the result would be `"Firmicutes"` too, there is no point in calculating the result. And because this package 'knows' all phyla of all known microorganisms (according to ITIS), it can just return the initial value immediately.
+Of course, when running `mo_phylum("Firmicutes")` the function has zero knowledge about the actual microorganism, namely *S. aureus*. But since the result would be `"Firmicutes"` too, there is no point in calculating the result. And because this package 'knows' all phyla of all known bacteria (according to the Catalogue of Life), it can just return the initial value immediately.
### Results in other languages
When the system language is non-English and supported by this `AMR` package, some functions take a little while longer:
-```{r, eval = FALSE}
+```{r}
mo_fullname("CoNS", language = "en") # or just mo_fullname("CoNS") on an English system
-# "Coagulase Negative Staphylococcus (CoNS)"
mo_fullname("CoNS", language = "fr") # or just mo_fullname("CoNS") on a French system
-# "Staphylococcus à coagulase négative (CoNS)"
microbenchmark(en = mo_fullname("CoNS", language = "en"),
de = mo_fullname("CoNS", language = "de"),
@@ -204,15 +154,6 @@ microbenchmark(en = mo_fullname("CoNS", language = "en"),
pt = mo_fullname("CoNS", language = "pt"),
times = 10,
unit = "ms")
-# Unit: milliseconds
-# expr min lq mean median uq max neval
-# en 6.093583 6.51724 6.555105 6.562986 6.630663 6.99698 100
-# de 13.934874 14.35137 16.891587 14.462210 14.764658 43.63956 100
-# nl 13.900092 14.34729 15.943268 14.424565 14.581535 43.76283 100
-# es 13.833813 14.34596 14.574783 14.439757 14.653994 17.49168 100
-# it 13.811883 14.36621 15.179060 14.453515 14.812359 43.64284 100
-# fr 13.798683 14.37019 16.344731 14.468775 14.697610 48.62923 100
-# pt 13.789674 14.36244 15.706321 14.443772 14.679905 44.76701 100
```
Currently supported are German, Dutch, Spanish, Italian, French and Portuguese.