MLSP 2009 Data Analysis Competition: Where's Wally?

Competition Committee: Vince D. Calhoun, Kenneth E. Hild II

Last Updated:	April 3, 2009.
Deadline:	Submissions must be emailed to hildk@bme.ogi.edu on or before April 22, 2009.
Goal:	Develop and submit a machine learning algorithm that automatically finds Wally, the pre-specified human target. The team that produces the largest score, defined below, will be deemed the winner of the competition. The teams representing the three best entries will receive a cash travel award and will be invited to publish a description of their approach in the conference proceedings. Anyone can participate in the competition. However, in order for a team to be eligible to receive the travel award and/or to be invited to publish their method in the proceedings at least one team member must register for the workshop and the method must employ some type of machine learning. The test data used for the competition will not be made available to the entrants. The total number of awards will be limited to the minimum of three and the number of unique entries divided by 2 (rounded towards 0). The decision of the judges is final. First place: 300 EUR + 1 publication Second place: 200 EUR + 1 publication Third place: 100 EUR + 1 publication
Updates:	If you wish to receive important updates on this year's competition via email then please send an email to hildk@bme.ogi.edu.
Training Data:	The training data may be downloaded from mlsp09TrainingData.mat. The training data is in Matlab format. It consists of a single (5 x 1) cell variable, inp, which contains three versions of a single image and two additional images. The sole human in the first image, Wally, is the designated target (Wally also appears in the third and fourth images).
Testing Methodology:	Once the submission deadline has passed the competition committee will run the submitted Matlab code and then estimate the score for each team. The score is determined using two values: one that is inversely related to completion time and one that measures recognition performance. The final score is given by, where C is the mean completion time per image measured in units of minutes (the code will be run on a 64-bit Windows XP machine), U is the expected utility, which is defined below, and U(min) and U(max) are respectively the minimum and maximum attainable values of expected utility (these values depend on N and the number of targets in the test set). The utility function determines the number of points awarded for each true positive (TP), false positive (FP), true negative (TN), and false negative (FN). The expected utility is given by, where the numbers of each type of event sum to N, which is the number of images in the test set, and where the individual utilities are given by, These values were chosen so that the expected utility associated with random guesses is approximately 0. These four categories are determined at the image level. Hence, for a given algorithm each image in the test set will be associated with either a true positive, false positive, true negative, or false negative. A true positive is associated with an image iff the probability of target for that image exceeds the user-defined threshold and the pixel selected for that image is contained in the target. Running the sample code below will illustrate the differences between the four types of events. Each of the N images in the test set is unique, unlike the training data. The resolution of some images is poorer than the other images. The prior probability that an image contains Wally is 0.60. He appears in different poses, in different types of lighting (e.g., indoor and outdoor), and in different locations within the images. Sometimes Wally is partly occluded, although his face is always visible and he always wears the same clothes, with the exception that he may or may not be wearing the blue scarf from one image to the next. Sometimes Wally is near the camera and sometimes he is far away. The mean size of Wally's head is (18 x 29) pixels squared and the minimum and maximum sizes are (9 x 14) and (36 x 54) pixels squared, respectively.
Submission:	A successful submission consists of (1) a 1-2 page description of the approach used, (2) Matlab code, and (3) at most one data file. The competition committee will write an overview document, which will be published in the proceedings, that will describe the competition and the data and will show the final results. It will also list the names of the team members that submitted the three best algorithms. The description of the method provided by each team needs to include the names of all team members and their associated host institution(s). It must also follow the MLSP submission format except that it is acceptable if the method has already been published, for sake of consistency it must use the mathematical notation below (notation not defined below can be selected as desired), the number of pages is limited to two, and the title should be, "MLSP 2009 Data Analysis Competition: Team # < team number >." This document should focus on the submitted algorithm since the overview document will describe the competition and the data. The Matlab code must adhere to the following requirements. The code must not issue any write commands. The code must only consist of regular uncompiled Matlab code (e.g., P-code, mex files, and compiled code are forbidden). Each team's code is expected to accept a single (N x 1) cell input variable, which has the same format as the training data. Namely, all data are represented by 8-bit unsigned integers and the ith element of the cell variable is either an RGB color image or a grayscale image. The size of the images can differ from one image to the next. Each team's code is expected to output three variables. The first is an (N x 2) matrix that gives the x-coordinates (first column) and the y-coordinates (second column) of the single pixel of each of the N images that is most likely to be contained within the target given that a target exists in the image. The second output variable is an (N x 1) vector of real numbers. The third output variable is a (1 x 1) threshold. An algorithm indicates that image i contains a target by choosing a value for the ith element of the second output variable and a value for the third output variable such that that the former is larger than the latter. The ith element of the second output variable can represent, e.g., the probability that pixel (x,y) of image i is contained in the target, which is given by, where P(T) = 0.6 is the prior probability. The platform that will be used for testing has the following Matlab toolboxes: Control System, Curve Fitting, Fuzzy Logic, System Identification, Image Processing, Neural Network, Optimization, Statistics, Symbolic Math, and Wavelet.
Sample Code:	The sample code, which follows the required format described above, may be downloaded from Find_Wally.m. This algorithm is not based on machine learning principles. It is included only to help potential entrants become acquainted with the testing procedure. Run this code by loading the training data and calling the provided function, >> load mlsp09TrainingData >> [xy,prob,thr] = Find_Wally(inp); Based on the values of prob and the threshold, thr, the sample algorithm determines that Wally is only in the first, fourth, and fifth images. Wally actually appears in the first, third, and fourth images. The first through fifth images respectively represent a true positive, true negative, false negative, false positive, and false positive. Notice that the fourth image represents a false positive since prob(4) > thr and the selected pixel is not contained in the target, whereas the fifth image is a false positive because prob(5) > thr and there is no target in this image. The expected utility for the sample code, based on the training data, is 5(1)-0.1(2)+1(1)-0.5(1) = 5.3.