1function hl = perf_profile(H,gate,logplot) 2% This subroutine produces a performance profile as described in: 3% 4% Benchmarking Derivative-Free Optimization Algorithms 5% Jorge J. More' and Stefan M. Wild 6% SIAM J. Optimization, Vol. 20 (1), pp.172-191, 2009. 7% 8% The latest version of this subroutine is always available at 9% http://www.mcs.anl.gov/~more/dfo/ 10% The authors would appreciate feedback and experiences from numerical 11% studies conducted using this subroutine. 12% 13% Performance profiles were originally introduced in 14% Benchmarking optimization software with performance profiles, 15% E.D. Dolan and J.J. More', 16% Mathematical Programming, 91 (2002), 201--213. 17% 18% The subroutine returns a handle to lines in a performance profile. 19% 20% H contains a three dimensional array of function values. 21% H(f,p,s) = function value # f for problem p and solver s. 22% gate is a positive constant reflecting the convergence tolerance. 23% logplot=1 is used to indicate that a log (base 2) plot is desired. 24% 25% Argonne National Laboratory 26% Jorge More' and Stefan Wild. January 2008. 27 28[nf,np,ns] = size(H); % Grab the dimensions 29 30% Produce a suitable history array with sorted entries: 31for j = 1:ns 32 for i = 2:nf 33 H(i,:,j) = min(H(i,:,j),H(i-1,:,j)); 34 end 35end 36 37prob_min = min(min(H),[],3); % The global minimum for each problem 38prob_max = H(1,:,1); % The starting value for each problem 39 40% For each problem and solver, determine the number of evaluations 41% required to reach the cutoff value 42T = zeros(np,ns); 43for p = 1:np 44 cutoff = prob_min(p) + gate*(prob_max(p) - prob_min(p)); 45 for s = 1:ns 46 nfevs = find(H(:,p,s) <= cutoff,1); 47 if (isempty(nfevs)) 48 T(p,s) = NaN; 49 else 50 T(p,s) = nfevs; 51 end 52 end 53end 54 55% Other colors, lines, and markers are easily possible: 56colors = ['b' 'r' 'k' 'm' 'c' 'g' 'y']; lines = {'-' '-.' '--'}; 57markers = [ 's' 'o' '^' 'v' 'p' '<' 'x' 'h' '+' 'd' '*' '<' ]; 58 59if (nargin < 3); logplot = 0; end 60 61% Compute ratios and divide by smallest element in each row. 62r = T./repmat(min(T,[],2),1,ns); 63 64% Replace all NaN's with twice the max_ratio and sort. 65max_ratio = max(max(r)); 66r(isnan(r)) = 2*max_ratio; 67r = sort(r); 68 69% Plot stair graphs with markers. 70hl = zeros(ns,1); 71for s = 1:ns 72 [xs,ys] = stairs(r(:,s),(1:np)/np); 73 74 % Only plot one marker at the intercept 75 if (xs(1)==1) 76 vv = find(xs==1,1,'last'); 77 xs = xs(vv:end); ys = ys(vv:end); 78 end 79 80 sl = mod(s-1,3) + 1; sc = mod(s-1,7) + 1; sm = mod(s-1,12) + 1; 81 option1 = [char(lines(sl)) colors(sc) markers(sm)]; 82 if (logplot) 83 hl(s) = semilogx(xs,ys,option1); 84 else 85 hl(s) = plot(xs,ys,option1); 86 end 87 hold on; 88end 89 90% Axis properties are set so that failures are not shown, but with the 91% max_ratio data points shown. This highlights the "flatline" effect. 92if (logplot) 93 axis([1 1.1*max_ratio 0 1]); 94 twop = floor(log2(1.1*max_ratio)); 95 set(gca,'XTick',2.^[0:twop]) 96else 97 axis([1 1.1*max_ratio 0 1]); 98end 99