d <- data.frame(x=rnorm(10), group=c(1,1,1,2,2,2,3,3,3,3))
d
            x group
1   1.0089900     1
2   0.4854706     1
3   1.7083259     1
4  -1.3362274     2
5   1.4905259     2
6   1.6451234     2
7   1.0361174     3
8   0.2369341     3
9  -2.0043264     3
10  1.4361718     3
library(caret)
d[createDataPartition(d$group, p=3/10)$Resample1,]
            x group
3   1.7083259     1
4  -1.3362274     2
8   0.2369341     3
10  1.4361718     3

blah blah blah | Rscript -e 'summary(scan(file("stdin")))'

m <- 10
k <- 3
d <- data.frame(x=factor(rep(1:k,m)), y=rnorm(m*k), z=rep(1:m,each=k))
d$ymax <- d$y + 1
d$ymin <- d$y - 1

# pretty simple
library(ggplot2)
p <- ggplot(d, aes(x=x, y=y, ymin=ymin, ymax=ymax))
p + geom_pointrange() + theme_bw() + facet_wrap(~ z)

# messy
par(mfrow=c(3,4), mar=c(3,3,2,1))
for (i in 1:m) {
  with(d[d$z == i,], {
    plot(as.numeric(x), y, main=i, xlim=c(0,k+1), ylim=c(-3,3), pch=20, xaxt="n")
    axis(1,at=1:k,1:k)
    segments(as.numeric(x),ymin,as.numeric(x),ymax)
  })  
}

hang <- 0.07
hc <- hclust(dist)
plot(hc)
pt.heights <- c(hc$height[hc$merge[,1] < 0],hc$height[hc$merge[,2] < 0])[order(-1 * c(hc$merge[,1][hc$merge[,1] < 0],hc$merge[,2][hc$merge[,2] < 0]))]
points(1:length(hc$order), pt.heights[hc$order] - hang)

In looking at sequential data (e.g. time-series or genomic data), any inference comparing different sequences needs to take into account local correlations within a sequence. For example, you might want to know how often is it raining in two cities at the same time, and if this is more than expected by chance. But it is more likely to rain on a given day if it was raining the day before, and this dependence will change the distribution of overlap expected by chance. In stochastics, this is a question of whether the process is ‘stationary‘.

One way out of the problem of estimating the distribution of overlap of two process by chance is the block bootstrap. Instead of randomly shifting features in the sequence (what I call naive permutation), you randomly build new sequences from large blocks of the original sequence. Then a distribution can be formed of overlap of features by chance. Here is a single bootstrap sample (top sequence) constructed in this manner from the data (bottom sequence).

Here are histograms demonstrating various ways of estimating the null distribution of overlaps between two sequences, with the true null on top (the clusters of features are of size 20). The block bootstrap can do a much better job of estimating the mean and variance of the null distribution. Knowing how large of a block to define is another problem, and Politis and Romano (below) explore the effect of using randomly sized blocks over fixed size blocks.

a reference for this problem in genomic inference is: Peter Bickel Boley N, Brown JB, Huang H and Zhang NR, Non-Parametric Methods for Genomic Inference, 2010, http://www.stat.berkeley.edu/~bickel/Bickel%20et%20al%202010%20AAS.pdf

and a more general reference is Dimitris N. Politis and Joseph P. Romano, The Stationary Bootstrap, Journal of the American Statistical Association, Vol. 89, No. 428 (Dec., 1994), pp. 1303-1313, http://www.jstor.org/discover/10.2307/2290993?uid=3737864&uid=2129&uid=2&uid=70&uid=4&sid=21101109652181

The R code for this example is here.

Here is some code I have used to look into this. I make a mixture of Gaussian data in n dimensions, where some dimensions are better than others at separating the classes. I do PCA on the first 2/3 of the rows, train a logistic regression on the first 1/3, and compare predictions on the middle 1/3 and the last 1/3:

compare <- function(m = 300, n = 10, how.much.signal = 1, k = 1) {
  # some random gaussian data
  x <- matrix(rnorm(m*n), nrow=m, ncol=n)
  # add some random noise to half the rows
  y <- rep(0:1, times=m/2)
  ramped.noise <- sapply(1:n/n * how.much.signal, function(i) rnorm(m/2,i,1))
  x[y == 1,] <- x[y == 1,] + ramped.noise
  # find the top principal compent using the first 2/3 rows
  pc <- prcomp(x[1:(m*2/3),])
  # train a model on first 1/3
  data <- data.frame(y=y, (x %*% pc$rotation[,1:k]))
  fit <- glm(y ~ ., data=data[1:m/3,], family="binomial")
  # predict on all data
  prediction <- predict(fit, newdata=data) > 0
  # test on middle 1/3
  middle.third <- (m/3 + 1):(m*2/3)
  error.middle <- mean(y[middle.third] != prediction[middle.third])
  # test on last 1/3
  last.third <- (m*2/3 + 1):m
  error.last <- mean(y[last.third] != prediction[last.third])
  return(error.last - error.middle)
}

repeated.comparison <- replicate(1000,compare())

Confidence intervals for 1000 differences between the mean misclassification rates:

> t.test(repeated.comparison)$conf.int
[1] -0.0073857758  0.0002057758
attr(,"conf.level")
[1] 0.95