Please use this identifier to cite or link to this item:
Full metadata record
DC FieldValueLanguage
dc.contributor.authorChristie, Lee Aen_UK
dc.contributor.authorBrownlee, Alexanderen_UK
dc.contributor.authorWoodward, John Ren_UK
dc.description.abstractBenchmarks are important to demonstrate the utility of optimisation algorithms, but there is controversy about the practice of benchmarking; we could select instances that present our algorithm favourably, and dismiss those on which our algorithm under-performs. Several papers highlight the pitfalls concerned with benchmarking, some of which concern the context of the automated design of algorithms, where we use a set of problem instances (benchmarks) to train our algorithm. As with machine learning, if the training set does not reflect the test set, the algorithm will not generalize. This raises some open questions concerning the use of test instances to automatically design algorithms. We use differential evolution, and sweep the parameter settings to investigate the practice of benchmarking using the BBOB benchmarks. We make three key findings. Firstly, several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances, possibly introducing unwanted bias to a resulting automatically designed algorithm. Secondly, the number of evaluations can have a large effect on the conclusion. Finally, a systematic sweep of the parameters shows how performance varies with time across the space of algorithm configurations. The data sets, including all computed features, the evolved policies, and their performances, and the visualisations for all feature sets, are available from
dc.publisherUniversity of Stirlingen_UK
dc.relationChristie LA, Brownlee A & Woodward JR (2018) Investigating Benchmark Correlations when Comparing Algorithms with Parameter Tuning (Detailed Experiments and Results). Not applicable. Stirling: University of Stirling.en_UK
dc.rightsAuthors retains copyright.en_UK
dc.subjectdifferential evolutionen_UK
dc.subjectcontinuous optimisationen_UK
dc.subjectparameter tuningen_UK
dc.subjectautomated design of algorithmsen_UK
dc.titleInvestigating Benchmark Correlations when Comparing Algorithms with Parameter Tuning (Detailed Experiments and Results)en_UK
dc.typeTechnical Reporten_UK
dc.contributor.sponsorNot applicableen_UK
dc.type.statusAM - Accepted Manuscripten_UK
dc.contributor.funderEngineering and Physical Sciences Research Councilen_UK
dc.contributor.funderEngineering and Physical Sciences Research Councilen_UK
dc.description.notesWork funded by UK EPSRC [grants EP/N002849/1, EP/J017515/1]. Results obtained using the EPSRC funded ARCHIE-WeSt HPC [EPSRC grant EP/K000586/1].en_UK
dc.contributor.affiliationComputing Scienceen_UK
dc.contributor.affiliationComputing Scienceen_UK
dc.contributor.affiliationQueen Mary, University of Londonen_UK
dc.relation.funderprojectFAIME: A Feature based Framework to Automatically Integrate and Improve Metaheuristics via Examples.en_UK
dc.relation.funderprojectDAASE: Dynamic Adaptive Automated Software Engineeringen_UK
rioxxterms.apcnot requireden_UK
rioxxterms.typeTechnical Reporten_UK
local.rioxx.authorChristie, Lee A|0000-0001-8878-0344en_UK
local.rioxx.authorBrownlee, Alexander|0000-0003-2892-5059en_UK
local.rioxx.authorWoodward, John R|en_UK
local.rioxx.projectEP/N002849/1|Engineering and Physical Sciences Research Council|
local.rioxx.projectEP/J017515/1|Engineering and Physical Sciences Research Council|
Appears in Collections:Computing Science and Mathematics Technical Reports

Files in This Item:
File Description SizeFormat 
investigating-benchmark-correlations-techreport.pdfFulltext - Accepted Version986.56 kBAdobe PDFView/Open

This item is protected by original copyright

Items in the Repository are protected by copyright, with all rights reserved, unless otherwise indicated.

The metadata of the records in the Repository are available under the CC0 public domain dedication: No Rights Reserved

If you believe that any material held in STORRE infringes copyright, please contact providing details and we will remove the Work from public display in STORRE and investigate your claim.