Perform a two-sample or paired-sample randomization (permutation) test for location based on either means or medians.

twoSamplePermutationTestLocation(x, y, fcn = "mean", alternative = "two.sided", 
    mu1.minus.mu2 = 0, paired = FALSE, exact = FALSE, n.permutations = 5000, 
    seed = NULL, tol = sqrt(.Machine$double.eps))

Arguments

x

numeric vector of observations from population 1. Missing (NA), undefined (NaN), and infinite (Inf, -Inf) values are allowed but will be removed.

y

numeric vector of observations from population 2. Missing (NA), undefined (NaN), and infinite (Inf, -Inf) values are allowed but will be removed.

In the case when paired=TRUE, the argument y must have the same number of elements as the argument x.

fcn

character string indicating which location parameter to compare between the two groups. The possible values are fcn="mean" (the default) and fcn="median". This argument is ignored when paired=TRUE.

alternative

character string indicating the kind of alternative hypothesis. The possible values are "two.sided" (the default), "less", and "greater".

mu1.minus.mu2

numeric scalar indicating the hypothesized value of the difference between the means or medians. The default value is mu1.minus.mu2=0.

paired

logical scalar indicating whether to perform a paired or two-sample permutation test. The possible values are paired=FALSE (the default; indicates a two-sample permutation test) and paired=TRUE (indicates take differences of pairs and perform a one-sample permutation test).

exact

logical scalar indicating whether to perform the exact permutation test (i.e., enumerate all possible permutations) or simply sample from the permutation distribution. The default value is exact=FALSE.

n.permutations

integer indicating how many times to sample from the permutation distribution when exact=FALSE. The default value is n.permutations=5000. This argument is ignored when exact=TRUE.

seed

positive integer to pass to the R function set.seed. The default is seed=NULL, in which case the current value of .Random.seed is used. Using the seed argument lets you reproduce the exact same result if all other arguments stay the same.

tol

numeric scalar indicating the tolerance to use for computing the p-value for the two-sample permutation test. The default value is
tol=sqrt(.Machine$double.eps). See the DETAILS section below for more information.

Details

Randomization Tests
In 1935, R.A. Fisher introduced the idea of a randomization test (Manly, 2007, p. 107; Efron and Tibshirani, 1993, Chapter 15), which is based on trying to answer the question: “Did the observed pattern happen by chance, or does the pattern indicate the null hypothesis is not true?” A randomization test works by simply enumerating all of the possible outcomes under the null hypothesis, then seeing where the observed outcome fits in. A randomization test is also called a permutation test, because it involves permuting the observations during the enumeration procedure (Manly, 2007, p. 3).

In the past, randomization tests have not been used as extensively as they are now because of the “large” computing resources needed to enumerate all of the possible outcomes, especially for large sample sizes. The advent of more powerful personal computers and software has allowed randomization tests to become much easier to perform. Depending on the sample size, however, it may still be too time consuming to enumerate all possible outcomes. In this case, the randomization test can still be performed by sampling from the randomization distribution, and comparing the observed outcome to this sampled permutation distribution.

Two-Sample Randomization Test for Location (paired=FALSE)
Let \(\underline{x} = x_1, x_2, \ldots, x_{n1}\) be a vector of \(n1\) independent and identically distributed (i.i.d.) observations from some distribution with location parameter (e.g., mean or median) \(\theta_1\), and let \(\underline{y} = y_1, y_2, \ldots, y_{n2}\) be a vector of \(n2\) i.i.d. observations from the same distribution with possibly different location parameter \(\theta_2\).

Consider the test of the null hypothesis that the difference in the location parameters is equal to some specified value: $$H_0: \delta = \delta_0 \;\;\;\;\;\; (1)$$ where $$\delta = \theta_1 - \theta_2 \;\;\;\;\;\; (2)$$ and \(\delta_0\) denotes the hypothesized difference in the meansures of location (usually \(\delta_0 = 0\)).

The three possible alternative hypotheses are the upper one-sided alternative (alternative="greater") $$H_a: \delta > \delta_0 \;\;\;\;\;\; (3)$$ the lower one-sided alternative (alternative="less") $$H_a: \delta < \delta_0 \;\;\;\;\;\; (4)$$ and the two-sided alternative $$H_a: \delta \ne \delta_0 \;\;\;\;\;\; (5)$$ To perform the test of the null hypothesis (1) versus any of the three alternatives (3)-(5), you can use the two-sample permutation test. The two sample permutation test is based on trying to answer the question, “Did the observed difference in means or medians happen by chance, or does the observed difference indicate that the null hypothesis is not true?” Under the null hypothesis, the underlying distributions for each group are the same, therefore it should make no difference which group an observation gets assigned to. The two-sample permutation test works by simply enumerating all possible permutations of group assignments, and for each permutation computing the difference between the measures of location for each group (Manly, 2007, p. 113; Efron and Tibshirani, 1993, p. 202). The measure of location for a group could be the mean, median, or any other measure you want to use. For example, if the observations from Group 1 are 3 and 5, and the observations from Group 2 are 4, 6, and 7, then there are 10 different ways of splitting these five observations into one group of size 2 and another group of size 3. The table below lists all of the possible group assignments, along with the differences in the group means.

Group 1Group 2Mean 1 - Mean 2
3, 45, 6, 7-2.5
3, 54, 6, 7-1.67
3, 64, 5, 7-0.83
3, 74, 5, 60
4, 53, 6, 7-0.83
4, 63, 5, 70
4, 73, 5, 60.83
5, 63, 4, 70.83
5, 73, 4, 61.67
6, 73, 4, 52.5

In this example, the observed group assignments and difference in means are shown in the second row of the table.

For a one-sided upper alternative (Equation (3)), the p-value is computed as the proportion of times that the differences of the means (or medians) in the permutation distribution are greater than or equal to the observed difference in means (or medians). For a one-sided lower alternative hypothesis (Equation (4)), the p-value is computed as the proportion of times that the differences in the means (or medians) in the permutation distribution are less than or equal to the observed difference in the means (or medians). For a two-sided alternative hypothesis (Equation (5)), the p-value is computed as the proportion of times the absolute values of the differences in the means (or medians) in the permutation distribution are greater than or equal to the absolute value of the observed difference in the means (or medians).

For this simple example, the one-sided upper, one-sided lower, and two-sided p-values are 0.9, 0.2 and 0.4, respectively.

Note: Because of the nature of machine arithmetic and how the permutation distribution is computed, a one-sided upper p-value is computed as the proportion of times that the differences of the means (or medians) in the permutation distribution are greater than or equal to [the observed difference in means (or medians) - a small tolerance value], where the tolerance value is determined by the argument tol. Similarly, a one-sided lower p-value is computed as the proportion of times that the differences in the means (or medians) in the permutation distribution are less than or equal to [the observed difference in the means (or medians) + a small tolerance value]. Finally, a two-sided p-value is computed as the proportion of times the absolute values of the differences in the means (or medians) in the permutation distribution are greater than or equal to [the absolute value of the observed difference in the means (or medians) - a small tolerance value].

In this simple example, we assumed the hypothesized differences in the means under the null hypothesis was \(\delta_0 = 0\). If we had hypothesized a different value for \(\delta_0\), then we would have had to subtract this value from each of the observations in Group 1 before permuting the group assignments to compute the permutation distribution of the differences of the means. As in the case of the one-sample permutation test, if the sample sizes for the groups become too large to compute all possible permutations of the group assignments, the permutation test can still be performed by sampling from the permutation distribution and comparing the observed difference in locations to the sampled permutation distribution of the difference in locations.

Unlike the two-sample Student's t-test, we do not have to worry about the normality assumption when we use a permutation test. The permutation test still assumes, however, that under the null hypothesis, the distributions of the observations from each group are exactly the same, and under the alternative hypothesis there is simply a shift in location (that is, the whole distribution of group 1 is shifted by some constant relative to the distribution of group 2). Mathematically, this can be written as follows: $$F_1(t) = F_2(t - \delta), \;\; -\infty < t < \infty \;\;\;\;\; (6)$$ where \(F_1\) and \(F_2\) denote the cumulative distribution functions for group 1 and group 2, respectively. If \(\delta > 0\), this implies that the observations in group 1 tend to be larger than the observations in group 2, and if \(\delta < 0\), this implies that the observations in group 1 tend to be smaller than the observations in group 2. Thus, the shape and spread (variance) of the two distributions should be the same whether the null hypothesis is true or not. Therefore, the Type I error rate for a permutation test can be affected by differences in variances between the two groups.

Confidence Intervals for the Difference in Means or Medians
Based on the relationship between hypothesis tests and confidence intervals, it is possible to construct a two-sided or one-sided \((1-\alpha)100\%\) confidence interval for the difference in means or medians based on the two-sample permutation test by finding the values of \(\delta_0\) that correspond to obtaining a p-value of \(\alpha\) (Manly, 2007, pp. 18–20, 114). A confidence interval based on the bootstrap however, will yield a similar type of confidence interval (Efron and Tibshirani, 1993, p. 214); see the help file for boot in the R package boot.

Paired-Sample Randomization Test for Location (paired=TRUE)
When the argument paired=TRUE, the arguments x and y are assumed to have the same length, and the \(n1 = n2 = n\) differences \(y_i = x_i - y_i\), \(i = 1, 2, \ldots, n\) are assumed to be independent observations from some symmetric distribution with mean \(\mu\). The one-sample permutation test can then be applied to the differences.

Value

A list of class "permutationTest" containing the results of the hypothesis test. See the help file for permutationTest.object for details.

References

Efron, B., and R.J. Tibshirani. (1993). An Introduction to the Bootstrap. Chapman and Hall, New York, Chapter 15.

Manly, B.F.J. (2007). Randomization, Bootstrap and Monte Carlo Methods in Biology. Third Edition. Chapman & Hall, New York, Chapter 6.

Millard, S.P., and N.K. Neerchal. (2001). Environmental Statistics with S-PLUS. CRC Press, Boca Raton, FL, pp.426–431.

Author

Steven P. Millard (EnvStats@ProbStatInfo.com)

Note

A frequent question in environmental statistics is “Is the concentration of chemical X in Area A greater than the concentration of chemical X in Area B?”. For example, in groundwater detection monitoring at hazardous and solid waste sites, the concentration of a chemical in the groundwater at a downgradient well must be compared to “background”. If the concentration is “above” the background then the site enters assessment monitoring. As another example, soil cleanup at a Superfund site may involve comparing the concentration of a chemical in the soil at a “cleaned up” site with the concentration at a “background” site. If the concentration at the “cleaned up” site is “greater” than the background concentration, then further investigation and remedial action may be required. Determining what it means for the chemical concentration to be “greater” than background is a policy decision: you may want to compare averages, medians, 95'th percentiles, etc.

Hypothesis tests you can use to compare “location” between two groups include: Student's t-test, Fisher's randomization test (described in this help file), the Wilcoxon rank sum test, other two-sample linear rank tests, the quantile test, and a test based on a bootstrap confidence interval.

Examples

  # Generate 10 observations from a lognormal distribution with parameters 
  # mean=5 and cv=2, and and 20 observations from a lognormal distribution with 
  # parameters mean=10 and cv=2.  Test the null hypothesis that the means of the 
  # two distributions are the same against the alternative that the mean for 
  # group 1 is less than the mean for group 2. 
  # (Note: the call to set.seed allows you to reproduce the same data 
  # (dat1 and dat2), and setting the argument seed=732 in the call to 
  # twoSamplePermutationTestLocation() lets you reproduce this example by 
  # getting the same sample from the permutation distribution).

  set.seed(256) 
  dat1 <- rlnormAlt(10, mean = 5, cv = 2) 
  dat2 <- rlnormAlt(20, mean = 10, cv = 2) 

  test.list <- twoSamplePermutationTestLocation(dat1, dat2, 
    alternative = "less", seed = 732) 

  # Print the results of the test 
  #------------------------------
  test.list 
#> 
#> Results of Hypothesis Test
#> --------------------------
#> 
#> Null Hypothesis:                 mu.x-mu.y = 0
#> 
#> Alternative Hypothesis:          True mu.x-mu.y is less than 0
#> 
#> Test Name:                       Two-Sample Permutation Test
#>                                  Based on Differences in Means
#>                                  (Based on Sampling
#>                                  Permutation Distribution
#>                                  5000 Times)
#> 
#> Estimated Parameter(s):          mean of x =  2.253439
#>                                  mean of y = 11.825430
#> 
#> Data:                            x = dat1
#>                                  y = dat2
#> 
#> Sample Sizes:                    nx = 10
#>                                  ny = 20
#> 
#> Test Statistic:                  mean.x - mean.y = -9.571991
#> 
#> P-value:                         0.0012
#> 

  #Results of Hypothesis Test
  #--------------------------
  #
  #Null Hypothesis:                 mu.x-mu.y = 0
  #
  #Alternative Hypothesis:          True mu.x-mu.y is less than 0
  #
  #Test Name:                       Two-Sample Permutation Test
  #                                 Based on Differences in Means
  #                                 (Based on Sampling
  #                                 Permutation Distribution
  #                                 5000 Times)
  #
  #Estimated Parameter(s):          mean of x =  2.253439
  #                                 mean of y = 11.825430
  #
  #Data:                            x = dat1
  #                                 y = dat2
  #
  #Sample Sizes:                    nx = 10
  #                                 ny = 20
  #
  #Test Statistic:                  mean.x - mean.y = -9.571991
  #
  #P-value:                         0.001


  # Plot the results of the test 
  #-----------------------------
  dev.new()
  plot(test.list)

  #==========

  # The guidance document "Statistical Methods for Evaluating the Attainment of 
  # Cleanup Standards, Volume 3: Reference-Based Standards for Soils and Solid 
  # Media" (USEPA, 1994b, pp. 6.22-6.25) contains observations of 
  # 1,2,3,4-Tetrachlorobenzene (TcCB) in ppb at a Reference Area and a Cleanup Area.  
  # These data are stored in the data frame EPA.94b.tccb.df.  Use the 
  # two-sample permutation test to test for a difference in means between the 
  # two areas vs. the alternative that the mean in the Cleanup Area is greater.  
  # Do the same thing for the medians.
  #
  # The permutation test based on comparing means shows a significant differnce, 
  # while the one based on comparing medians does not.


  # First test for a difference in the means.
  #------------------------------------------

  mean.list <- with(EPA.94b.tccb.df, 
    twoSamplePermutationTestLocation(
      TcCB[Area=="Cleanup"], TcCB[Area=="Reference"], 
      alternative = "greater", seed = 47)) 

  mean.list
#> 
#> Results of Hypothesis Test
#> --------------------------
#> 
#> Null Hypothesis:                 mu.x-mu.y = 0
#> 
#> Alternative Hypothesis:          True mu.x-mu.y is greater than 0
#> 
#> Test Name:                       Two-Sample Permutation Test
#>                                  Based on Differences in Means
#>                                  (Based on Sampling
#>                                  Permutation Distribution
#>                                  5000 Times)
#> 
#> Estimated Parameter(s):          mean of x = 3.9151948
#>                                  mean of y = 0.5985106
#> 
#> Data:                            x = TcCB[Area == "Cleanup"]  
#>                                  y = TcCB[Area == "Reference"]
#> 
#> Sample Sizes:                    nx = 77
#>                                  ny = 47
#> 
#> Test Statistic:                  mean.x - mean.y = 3.316684
#> 
#> P-value:                         0.0212
#> 

  #Results of Hypothesis Test
  #--------------------------
  #
  #Null Hypothesis:                 mu.x-mu.y = 0
  #
  #Alternative Hypothesis:          True mu.x-mu.y is greater than 0
  #
  #Test Name:                       Two-Sample Permutation Test
  #                                 Based on Differences in Means
  #                                 (Based on Sampling
  #                                 Permutation Distribution
  #                                 5000 Times)
  #
  #Estimated Parameter(s):          mean of x = 3.9151948
  #                                 mean of y = 0.5985106
  #
  #Data:                            x = TcCB[Area == "Cleanup"]  
  #                                 y = TcCB[Area == "Reference"]
  #
  #Sample Sizes:                    nx = 77
  #                                 ny = 47
  #
  #Test Statistic:                  mean.x - mean.y = 3.316684
  #
  #P-value:                         0.0206

  dev.new()
  plot(mean.list)


  #----------

  # Now test for a difference in the medians.
  #------------------------------------------

  median.list <- with(EPA.94b.tccb.df, 
    twoSamplePermutationTestLocation(
      TcCB[Area=="Cleanup"], TcCB[Area=="Reference"], 
      fcn = "median", alternative = "greater", seed = 47)) 

  median.list
#> 
#> Results of Hypothesis Test
#> --------------------------
#> 
#> Null Hypothesis:                 mu.x-mu.y = 0
#> 
#> Alternative Hypothesis:          True mu.x-mu.y is greater than 0
#> 
#> Test Name:                       Two-Sample Permutation Test
#>                                  Based on Differences in Medians
#>                                  (Based on Sampling
#>                                  Permutation Distribution
#>                                  5000 Times)
#> 
#> Estimated Parameter(s):          median of x = 0.43
#>                                  median of y = 0.54
#> 
#> Data:                            x = TcCB[Area == "Cleanup"]  
#>                                  y = TcCB[Area == "Reference"]
#> 
#> Sample Sizes:                    nx = 77
#>                                  ny = 47
#> 
#> Test Statistic:                  median.x - median.y = -0.11
#> 
#> P-value:                         0.9366
#> 

  #Results of Hypothesis Test
  #--------------------------
  #
  #Null Hypothesis:                 mu.x-mu.y = 0
  #
  #Alternative Hypothesis:          True mu.x-mu.y is greater than 0
  #
  #Test Name:                       Two-Sample Permutation Test
  #                                 Based on Differences in Medians
  #                                 (Based on Sampling
  #                                 Permutation Distribution
  #                                 5000 Times)
  #
  #Estimated Parameter(s):          median of x = 0.43
  #                                 median of y = 0.54
  #
  #Data:                            x = TcCB[Area == "Cleanup"]  
  #                                 y = TcCB[Area == "Reference"]
  #
  #Sample Sizes:                    nx = 77
  #                                 ny = 47
  #
  #Test Statistic:                  median.x - median.y = -0.11
  #
  #P-value:                         0.936

  dev.new()
  plot(median.list)

  #==========

  # Clean up
  #---------
  rm(test.list, mean.list, median.list)
  graphics.off()