MLSP 2009 Data Analysis Competition: Where's Wally?Competition Committee: Vince D. Calhoun, Kenneth E. Hild II
|Last Updated:||April 3, 2009.|
|Deadline:||Submissions must be emailed to email@example.com on or before April 22, 2009.|
|Goal:||Develop and submit a machine learning algorithm that automatically finds Wally, the pre-specified human target. The team that produces the largest score, defined below, will be deemed the winner of the competition. The teams representing the three best entries will receive a cash travel award and will be invited to publish a description of their approach in the conference proceedings. Anyone can participate in the competition. However, in order for a team to be eligible to receive the travel award and/or to be invited to publish their method in the proceedings at least one team member must register for the workshop and the method must employ some type of machine learning. The test data used for the competition will not be made available to the entrants. The total number of awards will be limited to the minimum of three and the number of unique entries divided by 2 (rounded towards 0). The decision of the judges is final.|
|Updates:||If you wish to receive important updates on this year's competition via email then please send an email to firstname.lastname@example.org.|
|Training Data:||The training data may be downloaded from mlsp09TrainingData.mat. The training data is in Matlab format. It consists of a single (5 x 1) cell variable, inp, which contains three versions of a single image and two additional images. The sole human in the first image, Wally, is the designated target (Wally also appears in the third and fourth images).|
|Testing Methodology:||Once the submission deadline has passed the competition committee will run the submitted Matlab code and then estimate the score for each team. The score is determined using two values: one that is inversely related to completion time and one that measures recognition performance. The final score is given by,|
Each of the N images in the test set is unique, unlike the training data. The resolution of some images is poorer than the other images. The prior probability that an image contains Wally is 0.60. He appears in different poses, in different types of lighting (e.g., indoor and outdoor), and in different locations within the images. Sometimes Wally is partly occluded, although his face is always visible and he always wears the same clothes, with the exception that he may or may not be wearing the blue scarf from one image to the next. Sometimes Wally is near the camera and sometimes he is far away. The mean size of Wally's head is (18 x 29) pixels squared and the minimum and maximum sizes are (9 x 14) and (36 x 54) pixels squared, respectively.
|Submission:||A successful submission consists of (1) a 1-2 page description of the approach used, (2) Matlab code, and (3) at most one data file.|
The competition committee will write an overview document, which will be published in the proceedings, that will describe the competition and the data and will show the final results. It will also list the names of the team members that submitted the three best algorithms.
The description of the method provided by each team needs to include the names of all team members and their associated host institution(s). It must also follow the MLSP submission format except that it is acceptable if the method has already been published, for sake of consistency it must use the mathematical notation below (notation not defined below can be selected as desired), the number of pages is limited to two, and the title should be, "MLSP 2009 Data Analysis Competition: Team # < team number >." This document should focus on the submitted algorithm since the overview document will describe the competition and the data.
The platform that will be used for testing has the following Matlab toolboxes: Control System, Curve Fitting, Fuzzy Logic, System Identification, Image Processing, Neural Network, Optimization, Statistics, Symbolic Math, and Wavelet.
sample code, which follows the required format described above, may be downloaded from Find_Wally.m. This algorithm is not based on machine learning principles. It is included only to help potential entrants become acquainted with the testing procedure. Run this code by loading the training data and calling the provided function,|
>> load mlsp09TrainingData
>> [xy,prob,thr] = Find_Wally(inp);
Based on the values of prob and the threshold, thr, the sample algorithm determines that Wally is only in the first, fourth, and fifth images. Wally actually appears in the first, third, and fourth images. The first through fifth images respectively represent a true positive, true negative, false negative, false positive, and false positive. Notice that the fourth image represents a false positive since prob(4) > thr and the selected pixel is not contained in the target, whereas the fifth image is a false positive because prob(5) > thr and there is no target in this image. The expected utility for the sample code, based on the training data, is 5(1)-0.1(2)+1(1)-0.5(1) = 5.3.