You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2017-02-17-release-0.8.9.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ date: 2017-02-17
5
5
categories: page
6
6
---
7
7
8
-
Today, <ahref="http://iao.hfuu.edu.cn/index.php/news/32-optimizationbenchmarking-beta">version `0.8.9`</a>, the "<ahref="http://iao.hfuu.edu.cn/index.php/news/32-optimizationbenchmarking-beta">Patched Beta</a>" release of our {{ site.projectNameStyled }} software suite, was published. Compared to the <ahref="{{ site.baseurl }}/page/2016/07/17/release-0.8.7.html">previous release</a>, this new version comes much closer to our goal to reduce the workload of researchers and practitioners during the development of algorithms to allow them to spend more time on thinking.
8
+
Today, version `0.8.9`, the "Patched Beta" release of our {{ site.projectNameStyled }} software suite, was published. Compared to the <ahref="{{ site.baseurl }}/page/2016/07/17/release-0.8.7.html">previous release</a>, this new version comes much closer to our goal to reduce the workload of researchers and practitioners during the development of algorithms to allow them to spend more time on thinking.
9
9
10
10
Besides all the functionality offered by the previous releases, it introduces a new process for obtaining high-level conclusions about problem hardness and algorithm behaviors. This process takes the raw data from experiments together with meta-information about algorithm setups and problem instances as input. It applies a sequence of machine learning technologies, namely curve fitting, clustering, and classification, to find which features make a problem instance hard and which algorithm setup parameters cause which algorithm behavior. We just submitted an article about this new process for review.
11
11
@@ -23,4 +23,4 @@ The <a href="https://hub.docker.com/r/optimizationbenchmarking/evaluator-gui/">d
23
23
24
24
The <ahref="https://github.com/optimizationBenchmarking/evaluator-evaluator/releases/download/0.8.9/evaluator.jar">CLI</a>, i.e., the command line interface, contains only the pure evaluator component and is to be executed manually. This may make sense when running it as component of another, non-Java process. If you would want to use the software inside your Java program, you can just the libraries directly. If you want to know how to do that, <a href="mailto:{{ site.contactEmail }}>contact me</a>.
25
25
26
-
Anyway, both the GUI and the CLI come as stand-alone `jar`s and you can run them directly via `java -jar evaluatorGui.jar` (in case of the GUI) and `java -jar evaluator.jar` in case of the evaluator. In the dockerized GUI version, this is already automatically done for you.
26
+
Anyway, both the GUI and the CLI come as stand-alone `jar`s and you can run them directly via `java -jar evaluatorGui.jar` (in case of the GUI) and `java -jar evaluator.jar` in case of the evaluator. In the dockerized GUI version, this is already automatically done for you.
Copy file name to clipboardExpand all lines: _posts/2017-07-27-research-talk.md
+4-32Lines changed: 4 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,37 +5,9 @@ date: 2017-07-27
5
5
categories: page
6
6
---
7
7
8
-
<p>One of my fundamental research interests is how we can determine which <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=61&catid=23">optimization algorithm</a> is good for which problem.</p>
9
-
<p>Unfortunately, answering this question is quite complicated. For most practically relevant problems, we need to <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=25&catid=30">find a trade-off</a> between the <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=29&catid=30">(run)time</a> we can invest in getting a good solution against the quality of said solution. Furthermore, the performance of almost all algorithms cannot just be a described by single pair of "solution quality" and "time needed to get a solution of that quality". Instead, these (anytime) algorithms start with an initial (often not-so-good) guess about the solution and then improve it step-by-step. In other words, their runtime behavior can be described as something like a function relating solution quality to runtime. But not a real function, since a) many algorithms are randomized, meaning that they behave differently every time you use them, even with the same input data, and b) an algorithm will usually behave different on different instances of an optimization problem type.</p>
8
+
<p>One of my fundamental research interests is how we can determine which optimization algorithm is good for which problem.</p>
9
+
<p>Unfortunately, answering this question is quite complicated. For most practically relevant problems, we need to find a trade-off between the (run)time we can invest in getting a good solution against the quality of said solution. Furthermore, the performance of almost all algorithms cannot just be a described by single pair of "solution quality" and "time needed to get a solution of that quality". Instead, these (anytime) algorithms start with an initial (often not-so-good) guess about the solution and then improve it step-by-step. In other words, their runtime behavior can be described as something like a function relating solution quality to runtime. But not a real function, since a) many algorithms are randomized, meaning that they behave differently every time you use them, even with the same input data, and b) an algorithm will usually behave different on different instances of an optimization problem type.</p>
10
10
<p>This means that we need to do a lot of experiments: We need to apply an optimization algorithm multiple times to a given optimization problem instance in order to "average out" the randomness. Each time, we need to collect data about the whole runtime behavior, not just the final results. Then we need to do this for multiple instances with different features in order to learn about how, e.g., the scale of a problem influences the algorithm behavior. This means that we will quite obtain a lot of data from many algorithm setups on many problem instances. The question that researchers face is thus "How can we extract useful information from that data?" How can we obtain information which helps us to improve our algorithms? How can we get data from which we can learn about the weaknesses of our methods so that we can improve them?</p>
11
-
<p>In this <ahref="http://iao.hfuu.edu.cn/images/publications/weise_2017_automating_research_work_in_optimization_slides.pdf"rel="alternate">research presentation</a>, I discuss my take on this subject. I introduce a process for automatically discovering the reasons why a certain algorithm behaves as it does and why a problem instance is harder for a set of algorithms than another. This process has already been implemented in our open source <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=17&catid=19">optimizationBenchmarking.org</a> framework.</p>
11
+
<p>In this research presentation, I discuss my take on this subject. I introduce a process for automatically discovering the reasons why a certain algorithm behaves as it does and why a problem instance is harder for a set of algorithms than another. This process has already been implemented in our open source optimizationBenchmarking.org framework.</p>
12
12
<h3>Talk Abstract</h3>
13
-
<p>In the fields of heuristic optimization and machine learning, experimentation is the way to assess the performance of an algorithm setup and the hardness of problems. Good experimentation is complicated. Most algorithms in the domain are anytime algorithms, meaning they can improve their approximation quality over time. This means that one algorithm may initially perform better than another one but <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=26&catid=30" rel="alternate">converge to worse solutions in the end</a>. Instead of single final results, the whole runtime behavior of algorithms needs to be compared (and runtime may be <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=29&catid=30" rel="alternate">measured in multiple ways</a>). We do not just want to know which algorithm performs best and which problem is the hardest ― a researcher wants to know <em>why</em>. We introduce a process which can 1) automatically model the progress of algorithm setups on different problem instances based on data collected in experiments, 2) use these models to discover clusters of algorithm (or problem instance) behaviors, and 3) propose reasons why a certain algorithm setup (or problem instance) belongs to a certain algorithm (or problem instance) behavior cluster. These high-level conclusions are presented in form of decision trees relating algorithm parameters (or instance features) to cluster ids. We emphasize the duality of analyzing algorithm setups and problem instances. Our process is implemented as <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=17&catid=19" rel="alternate">open source software</a> and tested in two case studies, on the Maximum Satisfiability Problem and the <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=16&catid=19">Traveling Salesman Problem</a>. Besides its basic application to <a href="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=60&catid=24" rel="alternate">raw experimental data</a>, yielding clusters and explanations of "quantitative" algorithm behavior, our process also allows for "qualitative" conclusions by feeding it with data which is normalized with problem features or algorithm parameters. It can also be applied recursively, e.g., to further investigate the behavior of the algorithms in the cluster with the best-performing setups on the problem instances belonging to the cluster of hardest instances. Both use cases are investigated in the case studies.</p>
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=77&catid=31">Ludwig-Maximilians-Universität Munich</a> in Munich (München), Germany</li>
17
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=76&catid=31">Johannes Gutenberg University Mainz</a> in Mainz, Germany</li>
18
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=75&catid=31">University of Cologne</a> in Köln (Cologne), Germany</li>
19
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=74&catid=31">University of Leipzig</a> in Leipzig, Germany</li>
20
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=73&catid=31">University of Applied Sciences Zittau/Görlitz</a> in Görlitz, Germany</li>
21
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=72&catid=31">Otto von Guericke University Magdeburg</a> in Magdeburg, Germany</li>
22
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=71&catid=31">Clausthal University of Technology</a> in Clausthal-Zellerfeld, Germany</li>
23
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=70&catid=31">Technische Universität Ilmenau</a> in Ilmenau, Germany</li>
24
-
<li>at the <ahref="http://www.cb.hs-mittweida.de/webs/villmann/miwoci-2017.html?Size=0%3Fprint%3D1%3Fprint%3D1%3Fprint%3D1"rel="alternate">MiWoCI Workshop</a> at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=69&catid=31">University of Applied Sciences Mittweida</a> in Mittweida, Germany</li>
25
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=68&catid=31">Chemnitz University of Technology</a> in Chemnitz, Germany</li>
26
-
<li>at the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=67&catid=31">Friedrich Schiller University</a> in Jena, Germany</li>
27
-
</ul>
28
-
<h3>Further Reading</h3>
29
-
<ul>
30
-
<li>the <ahref="http://iao.hfuu.edu.cn/images/publications/weise_2017_automating_research_work_in_optimization_slides.pdf"rel="alternate">slides</a> of this research presentation</li>
31
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=25&catid=30">What is Optimization?</a> (a description what optimization means from the metaheuristic perspective)</li>
32
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=61&catid=23">Metaheuristic Optimization</a> (a course on optimization, with all material publicly available on the website)</li>
33
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=17&catid=19">Optimization Benchmarking</a> (the project in which we implement the above ideas)</li>
34
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=12&catid=18">Rigorous Performance Analysis of Algorithms</a> (our research direction dealing with algorithm performance analysis)</li>
35
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=26&catid=30">Are Black-Box Global Search Methods, such as Metaheuristics like Evolutionary Algorithms, better than Local Search?</a> (the question whether global optimization algorithms are better than local search methods, from the perspective of anytime algorithms)</li>
36
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=46&catid=24">Two Papers accepted at GECCO 2017</a> (one is about how to dynamically model the runtime behavior of an optimization algorithm)</li>
37
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=60&catid=24">From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking</a> (a paper on a data format that can represent all information necessary for the above analysis, as used in the <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=17&catid=19">optimizationBenchmarking.org</a> project)</li>
38
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=16&catid=19">TSP Suite</a> (the predecessor project of <ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=17&catid=19">optimizationBenchmarking.org</a>)</li>
39
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=29&catid=30">Measuring the Runtime of (Optimization) Algorithms</a> (a post describing the problems that can arise when we measure the runtime of algorithms, which is essential for collecting data on algorithm runtime behavior)</li>
40
-
<li><ahref="http://iao.hfuu.edu.cn/index.php?option=com_content&view=article&id=32&catid=24">New Beta Release of our optimizationBenchmarking.org Software for Automating Research in Optimization</a> (the release notes of our tool on automated algorithm performance evaluation)</li>
41
-
</ul>
13
+
<p>In the fields of heuristic optimization and machine learning, experimentation is the way to assess the performance of an algorithm setup and the hardness of problems. Good experimentation is complicated. Most algorithms in the domain are anytime algorithms, meaning they can improve their approximation quality over time. This means that one algorithm may initially perform better than another one but converge to worse solutions in the end. Instead of single final results, the whole runtime behavior of algorithms needs to be compared (and runtime may be measured in multiple ways). We do not just want to know which algorithm performs best and which problem is the hardest ― a researcher wants to know <em>why</em>. We introduce a process which can 1) automatically model the progress of algorithm setups on different problem instances based on data collected in experiments, 2) use these models to discover clusters of algorithm (or problem instance) behaviors, and 3) propose reasons why a certain algorithm setup (or problem instance) belongs to a certain algorithm (or problem instance) behavior cluster. These high-level conclusions are presented in form of decision trees relating algorithm parameters (or instance features) to cluster ids. We emphasize the duality of analyzing algorithm setups and problem instances. Our process is implemented as open source software and tested in two case studies, on the Maximum Satisfiability Problem and the Traveling Salesman Problem. Besides its basic application to raw experimental data, yielding clusters and explanations of "quantitative" algorithm behavior, our process also allows for "qualitative" conclusions by feeding it with data which is normalized with problem features or algorithm parameters. It can also be applied recursively, e.g., to further investigate the behavior of the algorithms in the cluster with the best-performing setups on the problem instances belonging to the cluster of hardest instances. Both use cases are investigated in the case studies.</p>
Copy file name to clipboardExpand all lines: _posts/2017-10-01-bocia18.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,4 +5,4 @@ date: 2017-10-01
5
5
categories: page
6
6
---
7
7
8
-
We announce the <ahref="http://iao.hfuu.edu.cn/bocia18">International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA)</a> taking place at the Tenth International Conference on Advanced Computational Intelligence (ICACI 2018) in March 29-31, 2018 in Xiamen [厦门], Fujian [福建省], China. <ahref="http://iao.hfuu.edu.cn/bocia18">http://iao.hfuu.edu.cn/bocia18</a>.
8
+
We announce the International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA) taking place at the Tenth International Conference on Advanced Computational Intelligence (ICACI 2018) in March 29-31, 2018 in Xiamen [厦门], Fujian [福建省], China.
Copy file name to clipboardExpand all lines: _posts/2018-01-01-bbdob-gecco18.md
+1-2Lines changed: 1 addition & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,5 +5,4 @@ date: 2018-01-01
5
5
categories: page
6
6
---
7
7
8
-
We announce the <ahref="http://iao.hfuu.edu.cn/bbdob-gecco18">Black-Box Discrete Optimization Benchmarking (BB-DOB@GECCO) Workshop</a> at the 2018 Genetic and Evolutionary Computation Conference (GECCO 2018) in July 15-19, 2018 in Kyoto, Japan. <ahref="http://iao.hfuu.edu.cn/bbdob-gecco18">http://iao.hfuu.edu.cn/bbdob-gecco18</a>.
9
-
8
+
We announce the Black-Box Discrete Optimization Benchmarking (BB-DOB@GECCO) Workshop at the 2018 Genetic and Evolutionary Computation Conference (GECCO 2018) in July 15-19, 2018 in Kyoto, Japan.
0 commit comments