Silent Spring Institute Developer Blog

Bayesian Binomial Test in R

Thu, 11 Jan 2018 00:00:00 +0000

Summary: in this post, I implemenent an R function for computing $ P(\theta_1 > \theta2) $, where $ \theta_1 $ and $ \theta_2 $ are beta-distributed random variables. This is useful for estimating the probability that one binomial proportion is greater than another.

I am working on a project in which I need to compare two binomial proportions to see if one is likely to be greater than the other. I wanted to do this in a Bayesian framework; fortunately, estimating binomial proportions is one of the first subjects taught to introduce Bayesian statistics (and statistics in general, for that matter) so the method is well established. Rasmus Bååth has a very nice article describing the Bayesian binomial test, and an estimation approach using JAGS.

I made the problem even simpler by using the fact that the beta distribution is the conjugate prior to the binomial distribution. That is, if the prior is $ \mathrm{beta}(\alpha, \beta) $ distributed, then the posterior after observing $ k $ successes in $ n $ trials is

$\mathrm{beta}(\alpha + k, \beta + n - k)$

If you want to start with a noninformative prior, then you can use the beta distribution with $ \alpha = \beta = 1$, which collapses to the uniform distribution. This makes the posterior

$\mathrm{beta}(1 + k, 1 + n - k)$

What I am interested is in comparing two binomial proportions, $ \theta_1 $ and $ \theta_2 $. I want to calculate

$P(\theta_1>\theta_2)$

We observe $ k_1 $ successes out of $ n_1 $ trials and estimate a posterior distribution for $ \theta_1 $, and $ k_2 $ successes out of $ n_2 $ trials for $ \theta_2 $. Then the probability in question is

$P(\mathrm{beta}(1 + k_1, 1 + n_1 - k_1) > \mathrm{beta}(1 + k_2, 1 + n_2 - k_2)$

So far, so textbook. At this point, the simplest way to get an estimate is to draw two sets of random numbers, distributed by the two respective beta distributions, and see how many times the first random numbers is greater than the second:

set.seed(1)

k1 <- 5 # Five successes
k2 <- 1 # One success
n1 <- 10 # Ten trials
n2 <- 10 # Ten trials

b1 <- rbeta(10000, 1 + k1, 1 + n1 - k1)
b2 <- rbeta(10000, 1 + k2, 1 + n2 - k2)

# Probability estimate of P(theta_1 > theta_2)
sum(b1 > b2) / 10000

## [1] 0.9688

This works fine, but it would be nice to have an exact solution. I found several papers describing different approaches.

Nadarajah and Kotz (2007) derived a functional form for the distribution of $ P(\theta_1 - \theta_2) $ (and Chen and Luo fixed a typo in their work in 2011.) This approach is implemented in the tolerance package via the functions ddiffprop, pdiffprop, qdiffprop, and rdiffprop. The a1 and a2 parameters let you adjust the priors, where a1 = a2 = 1 is a uniform prior for both proportions.

library(tolerance)

# Probability that the first proportion (5 successes out of 10 trials)
# is greater than the second (1 success out of 10 trials)
pdiffprop(0,
          k1 = 5,
          k2 = 1,
          n1 = 10,
          n2 = 10,
          a1 = 1,
          a2 = 1,
          lower.tail = FALSE)

## [1] 0.9682663

The functions work well, but I found that they stop working for large values of $ n_1 $ or $ n_2 $. This is because several of the terms in the density function use the gamma function, and the R implementation returns infinity and throws a warning for any input larger than 171.

The second approach I found is described in Raineri et al. 2014, in which they derived a recursive formula for $ P(\theta_1 > \theta_2) $. I unrolled the recursion so it could handle large $ n $ without overflowing the stack, and implemented it in R. The first two functions, h and g, come directly from Raineri et al. and pdiff provides a wrapper for estimating differences in binomial proportions.

h <- function(a, b, c, d) {
  exp(lbeta(a + c, b + d) - (lbeta(a, b) + lbeta(c, d)))
}

# probability that beta(a, b) > beta(c, d)
g <- function(a, b, c, d) {
  stopifnot(a >= 1 && b >= 1 && c >= 1 && d >= 1)
  accum <- 0.5
  while (a != c || b != d) {
    if (a > c) {
      accum <- accum + h(a - 1, b, c, d) / (a - 1)
      a <- a - 1
    } else if (b > d) {
      accum <- accum - h(a, b - 1, c, d) / (b - 1)
      b <- b - 1
    } else if (c > a) {
      accum <- accum - h(a, b, c - 1, d) / (c - 1)
      c <- c - 1
    } else if (d > b) {
      accum <- accum + h(a, b, c, d - 1) / (d - 1)
      d <- d - 1
    }
  }
  accum
}

pdiff <- function(k1, k2, n1, n2, a1 = 1, a2 = 1) {
  g(1 + k1, a1 + n1 - k1, 1 + k2, a2 + n2 - k2)
}

# Probability that the first proportion (5 successes out of 10 trials)
# is greater than the second (1 success out of 10 trials)
pdiff(k1 = 5,
      k2 = 1,
      n1 = 10,
      n2 = 10)

## [1] 0.9682663

It can also handle large values for $ n_1 $ and $ n_2 $:

pdiff(k1 = 420,
      k2 = 400,
      n1 = 1000,
      n2 = 1000)

## [1] 0.8182686

The results from the simulation, tolerance package, and this new function all agree, which is a good sign. I ended up using the pdiff function for my project.

A baseball example

Let’s apply this to a simple example. Suppose we want to estimate an “ability score” for every baseball team, so we can see which teams are better than others. We model each team’s ability score as a binomial random variable representing the team’s proportion of wins to total games. By feeding in each team’s win and loss record for a season, we’ll estimate a posterior distribution that we can use to rank pairs of teams by how likely they are to have different ability scores.

First, we filter for the team records from 2016:

library(tidyverse)
library(Lahman)
library(knitr)

teams <- Teams %>%
  filter(yearID == 2016) %>%
  select(teamID, W, L) %>%
  mutate(teamID = as.character(teamID),
         P = paste0(round(W / (W + L) * 100), "%"))

And generate a dataframe that has the win and loss numbers from every pair of teams:

teams_compared <- expand.grid(teams$teamID, teams$teamID, stringsAsFactors = FALSE) %>%
  rename(team1 = Var1, team2 = Var2) %>%
  filter(team1 != team2) %>%
  left_join(teams, by = c(team1 = "teamID")) %>%
  left_join(teams, by = c(team2 = "teamID"), suffix = c("_1", "_2"))
  
head(teams_compared)

##   team1 team2 W_1 L_1 P_1 W_2 L_2 P_2
## 1   ATL   ARI  68  93 42%  69  93 43%
## 2   BAL   ARI  89  73 55%  69  93 43%
## 3   BOS   ARI  93  69 57%  69  93 43%
## 4   CHA   ARI  78  84 48%  69  93 43%
## 5   CHN   ARI 103  58 64%  69  93 43%
## 6   CIN   ARI  68  94 42%  69  93 43%

Now, we can use our pdiff function to calculate the probability that team 1’s win percentage is greater than team 2’s win percentage.

teams_compared <- teams_compared %>%
  mutate(p_1_greater = pmap(list(k1 = W_1, k2 = W_2, n1 = W_1 + L_1, n2 = W_2 + L_2), pdiff)) %>%
  unnest(p_1_greater)

Let’s see which teams are most likely to have a higher win percentage than other teams:

teams_compared %>%
  filter(P_1 > P_2) %>%
  arrange(-p_1_greater) %>%
  head()

##   team1 team2 W_1 L_1 P_1 W_2 L_2 P_2 p_1_greater
## 1   CHN   MIN 103  58 64%  59 103 36%   0.9999997
## 2   TEX   MIN  95  67 59%  59 103 36%   0.9999694
## 3   WAS   MIN  95  67 59%  59 103 36%   0.9999694
## 4   CHN   CIN 103  58 64%  68  94 42%   0.9999631
## 5   CHN   SDN 103  58 64%  68  94 42%   0.9999631
## 6   CHN   TBA 103  58 64%  68  94 42%   0.9999631

The Cubs (CHN), Rangers, and Nationals all have extremely high posterior probabilities of having a better win percentage than the Minnesota Twins.

Let’s see which teams are most likely to be the same:

teams_compared %>%
  filter(P_1 > P_2) %>%
  arrange(p_1_greater) %>%
  head()

##   team1 team2 W_1 L_1 P_1 W_2 L_2 P_2 p_1_greater
## 1   TEX   CLE  95  67 59%  94  67 58%   0.5186491
## 2   WAS   CLE  95  67 59%  94  67 58%   0.5186491
## 3   NYN   DET  87  75 54%  86  75 53%   0.5206036
## 4   SFN   DET  87  75 54%  86  75 53%   0.5206036
## 5   ARI   ATL  69  93 43%  68  93 42%   0.5257282
## 6   OAK   ATL  69  93 43%  68  93 42%   0.5257282

Texas and Cleveland had very similar records, so the posterior probability that Texas was better than Cleveland is only 52%.

A Bayesian approach to modelling censored data

Wed, 05 Jul 2017 00:00:00 +0000

For thise case, we can write Bayes formula as:

$p(\mu,\sigma|y^{\ast}) = \frac{p(y^{\ast}|\mu,\sigma)p(\mu,\sigma)}{p(y^{\ast})}$

The two components in the numerator are:

The probability of the data given a $ \mu $ and $ \sigma $, also called the likelihood function: $ p(y|\mu,\sigma) $
The probability of a given $ \mu $ and $ \sigma $, before seeing any data; also called the prior likelihood: $ p(\mu, \sigma) $

The denominator, $ p(y^{\ast}) $, is the total likelihood of the data integrated over all hypotheses (that is, over all values of $ \mu $ and $ \sigma $). This is the part of the formula that typically doesn’t have an analytical solution; in practice, numerical approximations need to be used.

Our job is to define the likelihood function and provide a prior distribution for $ \mu $ and $ \sigma $.

In the previous post, we derived the likelihood function for left-censored data:

$p\left(y^{\ast}|\mu,\sigma\right)=\prod\begin{cases} P\left(y^{\ast}_{i} | \mu,\sigma\right) & y^{\ast}_{i}>L\\ D\left(L | \mu,\sigma\right) & y^{\ast}_{i}=L \end{cases}$

where $ P $ is the log-normal probability density function and $ D $ is the log-normal cumulative density function.

The specification of a prior distribution depends on the application, and what is known about what reasonable values may be for $ \mu $ and $ \sigma $. In this case, let’s set a very weakly informative prior on both variables: $ \mu \sim Normal(0, 100) $ and $ \sigma \sim Normal(0, 100) $

A STAN program that encodes our likelihood function and prior specifications:

This program is written to look like the math, so it doesn’t use some special syntax that makes the code shorter, and it isn’t as computationally efficient as it code be.

data {
  int<lower=0> N;
  real L; // censoring limit
  real<lower=L> y[N];
  int cens[N]; // -1 if left-censored, 0 if not censored
}
parameters {
  real mu;
  real<lower=0> sigma;
}
model {
  // priors
  mu ~ normal(0, 100);
  sigma ~ normal(0, 100);

  // likelihood
  for(i in 1:N) {
    if(cens[i] == -1) {
      // left-censored
      target += normal_cdf(L | mu, sigma);
    }
    else {
      // not censored
      target += normal_pdf(y[i] | mu, sigma);
    }
  }
}

Unlocking Data in PDFs

Thu, 06 Apr 2017 00:00:00 +0000

Unfortunately, there is a lot of data released in on the web in the form of PDF files. Scraping data out of PFDs is much harder than scraping from a web page; web pages have structure, in the form of HTML, that you can usually leverage to extract structured data.

It isn’t hopeless, it’s just harder. Here are some of the tools and techniques that we’ve found useful in parsing data from PDFs.

`pdftotext`

pdftotext is a utility from the Xpdf project that converts PDFs to flat text files. It is easiest to install and use on unix based platforms, where it can be found in the poppler-utils package. There is also a windows port of Xpdf that I’ve used successfully.

pdftotext has several useful flags that effect how it parses its input:

-layout
-raw

tabula

Recently I’ve switched to Tabula for most of my PDF scraping needs. Tabula is a desktop application for extracting data from PDFs. I’ve found it to be more reliable than pdftotext. The only drawback is that it isn’t a command line program, so automating the scraping isn’t as easy as pdftotext. On the other hand, you can visually select the parts of the PDFs you’d like to scrape, which is useful for one-off jobs.

Tracking Packages in Excel

Thu, 23 Mar 2017 00:00:00 +0000

We’ve been shipping hundreds of packages as part of our Detox Me Action Kit project, and we wanted a way to track all of them without having to manually enter their tracking numbers on the UPS website. Through the UPS REST API and some VBA hacking, we were able to retrieve tracking information directly from Excel.

Dependencies

Download the VBA-JSON library and follow the instructions to reference it ni your spreadsheet.
An UPS account and a UPS API key, which you can request on their Developer Kit page.

The function works by submitting a request, formatted in JSON, to the UPS REST API tracking endpoint. The response, also in JSON, lists the events (arrivals and departures) linked to the package. The function returns the status of the most recent event. If no events are listed, then it returns “Not Shipped”. If the response has a fault or error, then “Error” is returned.

Function UPS(Tracking, Key, Username, Password)
    Dim MyRequest As Object
    
    Dim RequestContents As String
    RequestContents = "{"
    RequestContents = RequestContents & "'UPSSecurity': {"
    RequestContents = RequestContents & "    'UsernameToken': {"
    RequestContents = RequestContents & "        'Username': '" & Username & "',"
    RequestContents = RequestContents & "        'Password': '" & Password & "'"
    RequestContents = RequestContents & "    },"
    RequestContents = RequestContents & "   'ServiceAccessToken': {"
    RequestContents = RequestContents & "       'AccessLicenseNumber': '" & Key & "'"
    RequestContents = RequestContents & "   }"
    RequestContents = RequestContents & "},"
    RequestContents = RequestContents & "'TrackRequest': {"
    RequestContents = RequestContents & "   'Request': {"
    RequestContents = RequestContents & "       'RequestOption': '15',"
    RequestContents = RequestContents & "       'TransactionReference': {"
    RequestContents = RequestContents & "           'CustomerContext': 'Excel Package Tracker'"
    RequestContents = RequestContents & "        }"
    RequestContents = RequestContents & "   },"
    RequestContents = RequestContents & "   'InquiryNumber': '" & Tracking & "',"
    RequestContents = RequestContents & "   'TrackingOption':'02'"
    RequestContents = RequestContents & "}}"
    
    Set MyRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
    MyRequest.Open "POST", "https://onlinetools.ups.com/rest/Track"
    
    MyRequest.Send RequestContents
    
    Dim Json As Object
    Set Json = JsonConverter.ParseJson(MyRequest.ResponseText)
    
    Debug.Print JsonConverter.ConvertToJson(Json)
    
    If Json.Exists("Fault") Or Json.Exists("Error") Then
        UPS = "Error"
    Else
        Dim Package As Object
        Set Package = Json("TrackResponse")("Shipment")("Package")
        
        If TypeOf Package("Activity") Is Collection Then
            UPS = Package("Activity")(1)("Status")("Description")
        Else
            If Package("Activity").Exists("Status") Then
                UPS = "Not Shipped"
            End If
        End If
    End If
End Function

You can call the UPS function from an Excel formula:

=UPS("tracking number", "API key", "UPS username", "UPS password")

Happy tracking!

NHANES made simple with RNHANES

Tue, 22 Nov 2016 00:00:00 +0000

Scientists spend a lot of time “munging” data. Finding, cleaning, and managing datasets can take up the majority of the time it takes to complete an analysis. Tools that make the munging process easier can save scientists a lot of time.

We are tackling a small part of this problem in the context of the CDC’s National Health and Nutrition Examination Survey, a national study conducted every two years that gathers data on American’s health, nutrition, and exposure to environmental chemicals. Data from NHANES is used widely by environmental health scientists to study the U.S. public’s exposure to chemicals.

Figuring out how to set up a proper analysis of NHANES data can be tricky if you’re starting from scratch. That’s why we wrote RNHANES, an R package that makes it easy to download, analyze, and visualize NHANES data. RNHANES streamlines accessing the data, helps to set up analyses that correctly incorporate the study’s sample weights, and has built in plotting functions.

Let’s take the package for a spin by conducting a real-world analysis of NHANES data. Specifically, let’s characterize the U.S. population’s exposure to PFOA, a perfluo-alkyl substance (PFAS, also called “highly fluorinated compound”) that is linked to a number of adverse health effects.

Installation

A stable version of RNHANES is available in the CRAN package repository, and a development version is available on Github.

# Install stable version
install.packages("RNHANES")

# Install development version
devtools::install_github("silentspringinstitute/RNHANES")

Whichever version you install, you need to load the package. We will also be using the tidyverse package in this analysis because it is a handy wrapper package that automatically loads a number of other packages, like ggplot2 and dplyr, for you.

library(tidyverse)
library(RNHANES)

Downloading PFAS data

There is a comprehensive index online of the data available through NHANES. The index links to each file and the associated data dictionary. You can use the index to explore the data that is out there.

In our case, we are interested in PFASs. The latest available data on PFASs in NHANES is from the 2011-2012 survey cycle. You can download a data file easily through RNHANES:

pfas_data <- nhanes_load_data("PFC_G", "2011-2012", demographics = TRUE)

Summary Statistics

Now that we have the data, we can compute some summary statistics. A good first question to ask is how many people have a detectable level of PFOA in their blood samples, which we can find using the nhanes_detection_frequency function:

nhanes_detection_frequency(pfas_data, "LBXPFOA", "LBDPFOAL")

##       value     cycle begin_year end_year file_name  column weights_column
## 1 0.9973545 2011-2012       2011     2012     PFC_G LBXPFOA        WTSA2YR
##   comment_column                name
## 1       LBDPFOAL detection_frequency

99.7% of Americans have a detectable level of PFOA in their blood. The logical next step is to find out how much PFOA there is in people’s bodies. The 50th percentile (the median) level gives a estimate of the center of the population, and the 95th percentile an estimate of the levels in highly exposed individuals. With environmental chemicals, there can be large differences in exposure between the 95th and 50th percentiles.

quantiles <- nhanes_quantile(pfas_data, "LBXPFOA", "LBDPFOAL", quantiles = c(0.5, 0.95))

Plotting

We can plot the distribution as a histogram of values. This isn’t a very useful way of visualizing the data, because most environmental chemicals are log-normal. That is, the log transformation of the chemical levels roughly follows a normal distribution. We can see this for PFOA by applying a log transformation to the data.

nhanes_hist(pfas_data, "LBXPFOA", "LBDPFOAL")

nhanes_hist(pfas_data, "LBXPFOA", "LBDPFOAL", transform="log")

You can learn more about RNHANES on its Github page: http://github.com/silentspringinstitute/RNHANES.

Fitting censored log-normal data

Wed, 02 Nov 2016 00:00:00 +0000

Data are censored when samples from one or both ends of a continuous distribution are cut off and assigned one value. Environmental chemical exposure datasets are often left-censored: instruments can detect levels of chemicals down to a limit, underneath which you can’t say for sure how much of the chemical was present. The chemical may not have been present at all, or was present at a low enough level that the instrument couldn’t pick it up.

There are a number of methods of dealing with censored data in statistics. The crudest approach is to substitute a value for every censored value. This may be zero, the detection limit, the detection limit over the square root of two, or some other value. This method, however, biases the statistical results depending on what you choose to substitute.

A more nuanced alternative is to assume the data follows a particular distribution. In the case of environmental data, it often makes sense to assume a log-normal distribution. When we do this, while we may not know what the precise levels of every non-detect, we can at least assume that they follow a log-normal distribution.

In this article, we will walk through fitting a log-normal distribution to a censored dataset using the method of maximum likelihood.

Fitting a censored log-normal distribution

First, we need to generate a contrived dataset. Here we take 10000 samples from a log-normal distribution with a log-mean of 5 and log-standard deviation of 1; call those values $ y $. Any value below $ L = 50 $ will be censored. Let $ y^{\ast} $ be the censored dataset in which every censored point is fixed to the censoring limit. Formally, $ y^{\ast} $ is defined as:

$y^{\ast}=\begin{cases} y & y>L\\ L & y\leq L \end{cases}$

Here we generate a dataset in R:

library(ggplot2)
library(tibble)

L <- 50

data <- tibble(
  y = rlnorm(10000, meanlog = 5, sdlog = 1),
  censored = y < L,
  y_star = ifelse(censored, L, y)
)

ggplot(data, aes(x = y_star)) +
  geom_histogram(bins = 150) +
  scale_x_continuous(trans = "log") +
  xlab("log(y_star)")

The spike on the left represents the censored values. The detection frequency, or the proportion of detected values, is 0.8669.

First, let’s do a better job of defining the problem we’re trying to solve. We set out to fit a log-normal distribution to some censored data. Our fundamental problem here is that we want to find the right parameters (the mean and standard deviation) for a log-normal distribution that “fits”” the observed data. But how do you tell what the “right” parameters are, or what the best “fit” is? The maximum likelihood approach is to say “the best distribution is the one that has the highest likelihood of generating the data we’ve observed.” To actually find those parameters, you use an optimization algorithm that tries a bunch of different parameters and hones in on the one that defines a distribution with the highest likelihood of generating the observed data.¹

Let’s develop some intuition about how we are going to handle this censored data. With censored data, we don’t know the exact value of each datapoint that is below a censoring limit. The true value could be anywhere below the limit. To deal with this, we assume that the entire dataset is log-normally distributed, so even if we don’t know the exact value of the censored data, we know something about how those points are distributed.

Specifically, let $ P(x \vert \mu, \sigma) $ and $ D(x \vert \mu, \sigma) $ be the log-normal probability density function (PDF) and cumulative density function (CDF), respectively, taking as parameters a mean $ \mu $ and standard deviation $ \sigma $. We seek to find values for $ \mu $ and $ \sigma $ that maximize the likelihood function $ L(y^{\ast} \vert \mu, \sigma) $. A key assumption is that that the measurements in $ y^{\ast} $ are independent.

If the point is above the censoring limit, then its likelihood is defined by the log-normal PDF. If it below the censoring limit, we don’t know where the real value was exactly; all we know is that it’s less than $ L $. As such, its likelihood is the cumulative probability of a value being below $ L $ in a log-normal distribution with mean $ \mu $ and standard deviation $ \sigma $.

The total likelihood is found by multiplying all the individual likelihoods.² Formally:

$L\left(y^{\ast},\mu,\sigma\right)=\prod\begin{cases} P\left(y^{\ast}_{i},\mu,\sigma\right) & y^{\ast}_{i}>L\\ D\left(L,\mu,\sigma\right) & y^{\ast}_{i}=L \end{cases}$

It’s often more convenient, especially for computers, to work with the log-likelihood function. It’s easy to get it, just take the log of both sides:

$\mathcal{L}\left(y^{\ast},\mu,\sigma\right)=\sum\begin{cases} \log P\left(y^{\ast}_{i},\mu,\sigma\right) & y^{\ast}_{i}>L\\ \log D\left(L,\mu,\sigma\right) & y^{\ast}_{i}=L \end{cases}$

The R translation uses the built in dlnorm and plnorm functions for the log-normal PDF and CDF, choosing which one to use based on whether the data point is censored. It also checks the parameters to make sure they make sense: a negative standard deviation is invalid, so the function rejects that parameter set.

logLikCensoredFun <- function(params) {
  mean <- params[1]
  sd <- params[2]

  if(sd < 0) return(NA)
  
  sum(ifelse(data$censored, 
             plnorm(L, meanlog = mean, sdlog = sd, log.p = TRUE),
             dlnorm(data$y_star, meanlog = mean, sdlog = sd, log = TRUE)))
}

We use the maxLik function from the maxLik³ package to perform the estimation. We need to supply starting values for the parameters, so let’s have it start with a mean of 0 and standard deviation of 1.

library(maxLik)

## Error in library(maxLik): there is no package called 'maxLik'

fit <- maxLik(logLik = logLikCensoredFun,
              start = c(mean = 0, sd = 1))

## Error in eval(expr, envir, enclos): could not find function "maxLik"

summary(fit)

## Error in summary(fit): object 'fit' not found

The fitted distribution gets pretty close to recovering the correct mean and standard deviation. Censored data loses information, so in general you shouldn’t expect to get perfect estimates here.

Let’s plot the original, uncensored data $ y $ against the distribution fitted to the censored data to see how they compare visually:

ggplot(data = data, aes(x = y)) +
  geom_histogram(aes(y = ..density..), bins = 150) +
  stat_function(fun = dlnorm,
                args = list(meanlog = fit$estimate['mean'], sdlog = fit$estimate['sd']),
                color = 'green')

## Error in layer(data = data, mapping = mapping, stat = StatFunction, geom = geom, : object 'fit' not found

The core idea with this overall approach is that we assume something about the distribution of the censored data points so that we can get more nuanced parameter estimates than if we replaced them with a constant, like zero or the detection limit.

Future posts will go into regression techniques with censored datasets.

In “generative” statistical modelling you assume that the data you have was generated by some definable process. The task of the modeller is to write down how they think the data was generated, this is your “model” of how the world works. Everything that follows (parameter fitting, inference) is based off of this core assumption. In our case, we assume the data was generated from a log-normal distribution; that’s our “model” of how the data came to be. As it happens, this is exactly where the data came from! ↩
This comes from the assumption that each measurement is independent. Recall that the probability of two independent events occurring is the product of the individual probabilities: $ P(A \, \mathrm{and} \, B) = P(A) \cdot P(B) $ ↩
Henningsen, Arne and Toomet, Ott (2011). maxLik: A package for maximum likelihood estimation in R. Computational Statistics 26(3), 443-458. DOI 10.1007/s00180-010-0217-1. ↩

ggplot2 axis limit gotchas

Tue, 01 Nov 2016 00:00:00 +0000

Setting axis limits in ggplot has behaviour that may be unexpected: any data that falls outside of the limits is ignored, instead of just being hidden. This means that if you apply a statistic or calculation on the data, like plotting a box and whiskers plot, the result will only be based on the data within the limits.

In other words, ggplot doesn’t “zoom in” on a part of your plot when you apply an axis limit, it recalculates a new plot with the restricted data.

For example, here is a boxplot without any axis limits:

library(ggplot2)
data(iris)

ggplot(iris, aes(x = Species, y = Petal.Length)) +
  geom_boxplot()

and here is the same one with a set Y axis limit:

ggplot(iris, aes(x = Species, y = Petal.Length)) +
  geom_boxplot() +
  ylim(c(1, 4))

## Warning: Removed 84 rows containing non-finite values (stat_boxplot).

Note how the box and whiskers are recalculated in this plot. To be fair, ggplot does warn you that it is removing rows!

If you want to zoom in without removing values, use coord_cartesian:

ggplot(iris, aes(x = Species, y = Petal.Length)) +
  geom_boxplot() +
  coord_cartesian(ylim = c(1, 4))

What do chemicals look like?

Wed, 26 Oct 2016 00:00:00 +0000

If you look at the structures of chemicals within a family, you can see striking similarities. Take PCBs, which are industrial chemicals that were banned in the 1970s, as an example:

We wanted a way to generate beautiful chemical structure visualizations, like the ones you see above, for use in our materials. We developed a lightweight tool called sdftosvg that does just that: it takes as input SDF files, downloadable from repositories like PubChem, and outputs SVG renderings.

Installation

sdftosvg is available as an NPM package.

$ npm install -g sdftosvg

Example

The chemical triclosan is used as an anti-microbial in soaps, oral care products, and other consumer products. The PubChem page for triclosan has an SDF file available that defines triclosan’s structure. We can convert the SDF to an svg using sdftosvg:

$ sdftosvg triclosan.sdf triclosan.svg

Full documentation for the project is available on the project’s Github page.