One vs Multiple Test Instances

We use standard 5-fold cross-validation to evaluate our BiC(RoBiC, ...) system: we first divided the data into 5 balanced folds FF = {F1, …, F5} then use the information in FF - { Fi} to find labels for each instance ri,j ∈ Fi. Now recall, however, that BiC(RoBiC, ...)'s first step involves finding the biclusters based on both (the nonlabel part of) FF - { Fi} and Fi --- ie on all of FF. This means the biclusters (and hence the classification) for r5,1 depends on r5,2, r5,3, ... r5,3, as well as F1, …, F4.

Does this make a difference? In particular, how does the scenario compare with the more "standard" version, where the label for r5,1 depends only on itself and F1, …, F4, but NOT r5,2, r5,3, ... r5,15.

To find out, we took 4/5 of the BreastCancer data as the training set D, which here has 61 instances. We then considered each of the remaining 15 elements R = { r1, r2, ... r15} one by one. Here, we used the set of instances Di = D ∪ { ri } to produce the set of k=30 biclusters, Bi = { Bi,1, Bi,2, ..., Bi,k} = RoBiC( Di, k).

Of course, each of these Bi bicluster sets can be very different from one another. We can allay some of our worries if we find that these 15 different bicluster sets are similar to one another, and also to the biclusters obtained using the full FF set of instances, B* = RoBiC( FF, k). Below we present two ways to measure these similarities, focussing on just the first three biclusters for each set --- ie, comparing the members of {Bi,1}i = {B1,1, B2,1, ..., B15,1} with one another and with B*1; then comparing {Bi,2}i with each other and with B*2; and finally dealing with {Bi,3}i and B*3. For notation: each bicluster Bi,j involves a particular set of genes Gi,j.

(See also UseOnlyTraining for another way to use only the training data.)

** Comparing αi,j with αi*: FMeasure **

To compare any pair of sets --- eg, G1,1 with G1* --- we can use (a variant of) F-measure index,
F(A, B) = Fmeasure(A, B) =
  2 × |A ∩ B|
    |A| + |B|
(It is easy to confirm that this corresponds to (2× Prec(A,B) × Recall(A,B))/(Prec(A,B) + Recall(A,B)) where Prec(A,B) = |A ∩ B|/|A| and where Recall(A,B) = |A ∩ B|/|B|.)

We therefore computed the 15 values F( Gi,1, G1*), associated with the first bicluster of each bicluster-set. This is graphed in the far left region in left plot in Figure 1 below, as a box-and-whisker plot (produced with Matlab's BOXPLOT). (This plot in corresponds to the 15 values of F( Gi,1, G1*) over the 15 single patient additions.) We see that the mean is around 0.85, and one standard deviation is only a few percent. The middle region in this graph corresponds to the second biclusters { F( Gi,2, G2*) }; and the far right to the third biclusters { F( Gi,3, G3*) }.
Figure 1: Box-and-Whisker plot of (left) F( Gi,1, G1*), F( Gi,2, G2*) and F( Gi,3, G3*); and (right) F( Gi,1, Gj,1) F( Gi,2, Gj,2) and F( Gi,3, Gj,3).

We also compared all  (15 2)  pairs F( Gi,1, Gj,1 ) pairs, for (i ≠ j). The left graph of the Figure shows those values, for the first, second and third biclusters. Notice the average F-score here is around 0.95 for both the first and 2nd biclusters.

** Membership Histograms across {αi,j}i for α = G, P, PG and j=1,2,3 **

Figure 2(left) presents a histogram of the patients in the 15 biclusters {Pi,1}i = {P1,1, P2,1, ..., P15,1}; that is, we see that 6 patients appeared in all 15 of the bicluster#1's, and that 3 appeared in only 1. (This is out of the total of 11 patients that appeared in any bicluster#1; ie, in ∪j Pj,1.) Figure 2(mid) (resp., Figure 2(right)) shows a histogram of the 17 patients in the 15 bicluster#2's ∪j Pj,2 (resp., 16 patients in the 15 bicluster#3's ∪j Pj,3). It is not suprising that slightly fewer, 5 patients, appear in all 15 of the 2nd biclusters, and only 4 in all 15 3rd biclusters.

Figure 3 deals with genes. We note that almost 300 genes (of around 400) appear in all 15 bicluster#1's, and around 800 (of 1000) genes in all 15 bicluster#2's.
Figure 2: Histogram of the number of patients in each of the 15 biclusters for the first bicluster (left), the second bicluster (center) and the third bicluster (right).
Figure 3: Histogram of the number of genes in each of the 15 biclusters for the first bicluster (left), the second bicluster (center) and the third bicluster (right).

** Comparing biclusters for each of B1i and biclusters containing all of the 15 instances (B1*, B3*, B3*) **

Figure 4: Histogram of the number of patients in each of the 15 biclusters and the bicluster containing all instances for the first bicluster (left), the second bicluster (center) and the third bicluster (right).
Figure 5: Histogram of the number of genes in each of the 15 biclusters and the bicluster containing all instances for the first bicluster (left), the second bicluster (center) and the third bicluster (right).