Final Evaluation and Ranking

We will integrate the submitted algorithm with an Android downloader program, generate the APK, and evaluate the algorithm performance using our online testing platform. We have selected 8 different Android devices to deploy all candidate APKs and run downloading tasks. For each task, all candidate APKs will be brought up sequentially for execution, and the adjusted speed score will be reported after the APK completes or fails the downloading task. We do not run more than one downloading task simultaneously in the same LAN.

We plan to run several rounds of tests and eliminate bottom ranking teams each round. For each round, we will run 30~50 downloading tasks on each Android device for all candidates. We rank all candidates based on the mean value of all collected adjusted speed scores. If two teams have close scores (within 3%), we determine their ranking based on the following rules:
1. The team with less failed tasks (zero score) ranks higher;
2. For each Android device, determine the winner of two teams based on the mean score of all tasks running on this device. The team wins more devices ranks higher;
3. For each downloading task, determine the winner of two teams based on the reported adjusted speed score. The team wins more tasks ranks higher;
4. The team with less standard deviation value ranks higher.

Final Result

We have received 13 final submissions but could only successfully compile 12 of them. We deployed all compiled APKs to 8 Android phones (Phones are physically located in different cities all over China. 4 of them use WiFi and 4 of them use 4G/LTE networks) for the final test.

Round 1:

Each APK ran 148 times and our server was able to record 128 scores for most teams. We include the demo we provided in the test as well. You can download all the data here. The last three teams were eliminated from this round.

Round 2:

The remaining 10 APKs (including DEMO) ran 240 tasks each and you can download all the data here. Top three teams advanced to the final round. Top three teams advanced to the final round.

Round 3:

Top three teams together with our own product version MPD algorithm ran 400 tasks each. You can download all the data here.

Since the scores of team xiaoxuesheng and team TXPlayer are too close (<3%), we ran further analysis according to the preset rules:

1. The team with less failed tasks (zero score) ranks higher. Both teams recorded 346 non zero scores in the final round. We did notice that team xiaoxuesheng has one zero score record while team TXPlayer has no zero score record. However, from the available dataset we were not able to determine whether the zero score was caused by device issues or algorithm itself. Thus we call it a tie.
2. For each Android device, determine the winner of two teams based on the mean score of all tasks running on this device. The team wins more devices ranks higher. Each team wins 4 devics, still a tie.

3. For each downloading task, determine the winner of two teams based on the reported adjusted speed score. The team wins more tasks ranks higher. In this round, the execution sequence for each task was set to random. We sorted the score records based on the timestamp and chose adjacent scores for head to head comparison. Team xiaoxuesheng wins 221 of 335 head to head comparisons.

Testing Result