Quantile point

r/100 is the quantile point corresponding to the r% quantile Xr.


Display Results

CLIPBOARD to MS WORD:

To view properly the exported table data from EXCALIBUR in MS Word, you should choose a non-proportional font like Courier New or LinePrinter, which the bastard has eliminated from MS Word 6. However, you can first copy to MS WordPad, and choose the LinePrinter fonts, copy again to the clipboard, then to MS Word. You probably will want to choose small size fonts.

Alternatively, copy first to Notepad, then to MS Word.

File / Export as space delimited:

This feature is for exporting to EXCEL where graphs may be made. This can be done as follows:

After copying to EXCEL, select the first column (the percentile points) and columns corresponding to a given variable for the selected experts. Choose XY scatter plot. The result will be the experts’ cumulative distribution functions for the given item. These functions will be shown with the percentile points on the horizontal axis.

Expert scores:

The expert scores can be saved as a text file, or copied to the clipboard.

Solutions:

The last run DM is always the current solution. The solution may be saved as a text file, or copied to the clipboard.

Bilinear Loss:

The Bilinear Loss window’s contents can be copied to the clipboard, or exported as a text file.

Range graphs:

Right click inside the windows to export text. Use a non-proportional font.


Preliminaries

A case consists of two ascii files with the same name and extensions .dtt and .rls. These files are created automatically when you create a case. They may be edited from any editor, but the exact spacing must be maintained. If a user is given only the *.dtt file and not the corresponding *.rls, then the user can edit the items and experts, but cannot view the realizations and cannot calculate decision makers. This is used in elicitation via EXCALIBUR.

Elicitation via Questionnaire:

When a file is created, you can export a questionnaire containing the item descriptors and elicitation quantiles. When the Items screen is active, choose export as text from the file menu.

Elicitation via EXCALIBUR:

Make a CASENAME.dtt file. On saving, the CASENAME.rls file will be created. Copy the CASENAME.dtt file as "CASENAME._EXPERTNAME.dtt" and distribute this to Mr. Expertname. Do not create a CASENAME_EXPERTNAME.rls file. The experts may then edit their names and fill in their assessments. They cannot view the realizations and cannot compute decision makers.

NOTE: we strongly discourage elicitation in the absence of a qualified analyst. The item descriptors will not in general suffice to clarify the uncertainties being elicited, and the analyst MUST be available for clarification.

Merging Assessments:

When a file is opened, any other *.dtt files can be merged with the open file. The operation is intended for experts who have added their own assessments to CASENAME_EXPERTNAME.dtt. Items in the second file that are not found in the first file, are appended to the items of the first file.

Data Flow:

Normal data flow when creating a new case is

1.enter items and realizations (if available)

2.enter expert names

3.enter expert assessments

Row/Cell mode:

In the Items and Experts fields, one can be in one of two modes:

·Cell mode: for entering and editing of the numerical data inside of a single cell

·Row mode: for editing of the table data row-wise (inserting, deleting, selecting, filtering, etc. of a row or a group of rows)

Row mode is entered by clicking on the row index or hitting F3 or from the selecting from the EDIT menu. F3 works as a toggle between the row and cell modes, also the F2 button can be used to switch to the cell edditing mode exclusively.

Cell width:

Cell width can be changed by dragging the column separators atop of the grid window

Closing a Case:

A case may be closed from the Windows menu, or by closing the Expert, Expert Scores or Expert Robustness window.


Bayesian updates

Bayesian updating in the present implementation is accomplished using a non- informative prior and a multinomial likelihood function. The number of possible outcomes is one plus the number of quantiles. If the n quantile points are arranged:

then the j-th outcome of the multinomial distribution corresponds to the realization falling between the [image] -th and [image] -th quantile. By definition, an expert assigns the j-th outcome the probability [image].

In effect, the Bayesian updating recalibrates the experts' assessments (conditional on a choice of scale and cut-off points for each variable). Updated 5%, 50% and 95% quantiles are computed using a minimum information fit to the updated quantiles.


Thurstone B

In this model it is assumed that for all i <> j.

rij = r,

|si - sj| << si.

Under these conditions we can derive

mi - mj = F^(-1)(%(ij))*(1/2(1-r)^0.5 *(si + sj)).

Since the units of the Xi are arbitrary, we can write this as

mi - mj = F^(-1)(%(ij))(si + sj).

which is equivalent to assuming rij = -1 in Thurstones law of comparative judgments. With N objects, there are N CHOOSE 2 such equations with 2N unknowns. For N > 5 there are more equations than unknowns. We find a least squares solution for the mi, unique up to a positive affine transformation.


Thurstone C

In this model it is assumed that sij = s. For all practical purposes this equivalent to assuming that rij = 0 and si = s/2^0.5. Filling these into Thurstone's law, and recalling that the Xi are determined only up to an affine transformation, Thurstone's law becomes simply:

mi - mj = F^(-1)(%(ij))

A least squares routine is used to determine the mi up to a positive affine transformation.


Optimized DM

The optimized DM results by choosing that significance level for which the global unnormalized weight of the DM is maximal. Optimization is meaningful for global and item weights. It is not meaningful for equal or user weights.


Quantile

If X is a continuous valued random variable, the r% quantile of the distribution of X is the smallest number Xr such that

Probability{X < Xr} = r/100.


User weights

Weights supplied by the user.


Calculating

Run time failures can be caused by

Non-positive assessments for a variable with the LOG scale,

·Non- increasing quantiles,

·Running a performance-based DM without any realizations.

Filtering:

Items and experts can be filtered out of a calculation. This is done from the EDIT menu or with the function keys F5 and F4. F5 toggles the Filtering on and off, and when filtering is enabled, F4 selects or deselects items for filtering.

NOTE: It is impossible to filter from the Assessments window.

User weights:

Leave one weight unspecified, and it will be fixed automatically so that the weights sum to one.

Chi2:

·OLD: reproduces the calculations in the DOS EXCALIBR by using a tabulated form of the CHI square distribution.

·NEW: uses a more general numerical routine to calculate tail values for the CHI-square distribution.

Inf:

·OLD: reproduces the DOS EXCALIBR algorithm for determining the intrinsic range (see Glossary) with log scale variables close to zero.

·NEW: uses a better algorithm, namely first convert to log scale and apply the overshoot rule as in the case of uniform variables.

NOTE: NEW is the same as OLD for UNI scale

DM optimization:

Must be OFF for equal and user weights, must be ON for global and item weights.

TIP: DM’s can be given names and run successively. Successive DMS are added to the output files.


Discrepancy

To perform discrepancy analysis, the relative information of each expert's assessment, per item, is compared with the DM's assessment for that item, and the relative information of the expert with respect to the DM is computed. These scores are averaged over all items. The average scores (which are proportional to the relative information of the respective joint distributions if all items are independent) are output. This enables the user to see which experts agree or disagree most with the DM. (Dis)agreement is not well predicted by an experts unnormalized weight.

In addition, the ratio of the largest/smallest relative information score per item is output. This enables the user to flag those items for which the experts assessment of uncertainty differs most.


Unnormalized weight

Letting a denote the significance level, M the number of seed variables, and 1A the indicator function for event A, the unnormalized (global) weight for assessor j (either an expert or the DM) is computed as:

wj = C(j)*(1/M)*SUM(i=1..M)[I(f(i,j),g(I))*1{C(j)>a}]

In other words, the unnormalized weight displayed is always the global weight.

When using equal weights or user weights, a in the above expression is set equal to zero. When using item weights, a is chosen to optimize the DM's global unnormalized weight. However, the weights used to calculate the DM's assessments are not the unnormalized weights shown for the experts. Expert j's weight for item i is:

wj(i) = C(j)*I(f(i,j),g(i))*1{C(j) > a}.


Negative exponential lifetime model (Bradley-Terry)

In this model it is assumed that the object i is a component with exponential life distribution, and with failure rate Li, i = 1,…,N. For all pairs of objects, each subject is asked to determine which component has the greater probability of failing per unit time. In answering each such question, it is assumed that each subject performs a "mental experiment", observing which of two independent exponential components with failure rates Li, Lj fails first. Under these assumptions the probability pij that i fails before j is given by:

Lj

pij = ------- .

LI + LJ

The percentage of experts preferring i to j is taken to estimate pij. The above system of equations is overdetermined, and the principle of maximal likelihood may be used to solve for the Li's. Multiplying each failure rate in the above equation by a constant would leave the equations invariant, hence the resulting Li's are determined only up to a constant. If one of the Li is known, (reference value) the others may be determined from the maximum likelihood solution.

Note: The above system can be derived from other assumptions. For example, we might assume that the conditional probability pij of preferring cookie i to cookie j, given that one of i or j must be chosen, is proportional to the intrinsic goodness Li of cookie i, and that the same holds for j. Since these two probabilities must sum to one, the above system of equations results. The general method for solving this system was given by Bradley and Terry, who applied it to pairwise consumer testing of new products.


Intrinsic range

The expert's quantile assessments are used to fit a minimally informative distribution, relative to the background measure (see scale). For this minimization it is necessary that the set of possible values be restricted to a bounded interval. By default EXCALIBUR uses the "10% overshoot" rule: The smallest interval containing all assessments for a given item (plus the realization, if available) is overshot by 10% above and below. The expert's information scores are affected by the choice of the overshoot; making this overshoot very large tends to suppress differences in the experts' information scores, however, the effect is very slow.


References

Roger M. Cooke
„Experts in Uncertainty : Opinion and Subjective Probability in Science (Environmental Ethics and Science Policy Series)"

Oxford University Press, June 1991

This book is an extensive survey and critical examination of the literature on the use of expert opinion in scientific inquiry and policy making. The elicitation, representation, and use of expert opinion is increasingly important for two reasons: advancing technology leads to more and more
complex decision problems, and technologists are turning in greater numbers to "expert systems" and other similar artifacts of artificial intelligence. Cooke here considers how expert opinion is being used today, how an expert's uncertainty is or should be represented, how people do or should
reason with uncertainty, how the quality and usefulness of expert opinion can be assessed, and how the views of several experts might be combined. He argues for the importance of developing practical models with a transparent mathematic foundation for the use of expert opinion in science, and
presents three tested models, termed "classical," "Bayesian," and "psychological scaling." Detailed case studies illustrate how they can be applied to a diversity of real problems in engineering and planning.


Robustness (items)

Seed items are excluded from the analysis, one at a time and the resulting DMs are computed and compared, using the current parameter values under "RUN". The total relative information with respect to the background measure, the calibration and the total relative information with respect to the original DM are tallied.


Relative information

The relative information of probability vector s = (s1,...sn) with respect to p = (p1,...pn) is given by

I(s|p) = SUM(i=1..n) [si*ln(si/pi)].

If s is a sample distribution gotten from M independent samples from p, then 2MI(s,p) is asymptotically Chi square distributed with n-1 degrees of freedom.

Relative information is used as a Chi square test statistic in the measure of calibration, but this quantity is not displayed (one can recover this value by inverting the Chi square cumulative distribution function, and dividing by (2*M*power).

The relative information score displayed is the average over i of I(f(i,e),g(i)) where f(i,e) is the minimal information density function fitted to expert e's quantiles for item i, and g(i) is either the uniform or loguniform density function, depending on the scale of the item. Two scores are displayed, namely the average over all items and the average over the seed items.


Strictly proper scoring rules

Suppose an expert assesses that an uncertain quantity with outcomes 1,…,n has distribution p = p1,..pn. A scoring rule R assigns a score to this distribution on the basis of the realization, say i, R(p,i). If the expert "really believes" that the uncertain quantity follows a distribution q, then his expected score on stating p is Eq(R(p,i)). R is a strictly proper scoring rule for a single uncertainty quantity if

argmax(p) Eq(R(p,i)) = q.

In other words, the expert maximizes his expected score under R by, and only by, stating his true belief. The classical model is based on a generalization of this notion, whereby a score is associated with a SET of assessments and realizations. The variables in question are assumed for convenience to have the same range (1,...n). Let the set of assessments consists of M variables with assessed distribution p (see Calibration). The set of sample realizations may be represented with the sample distribution s (see Calibration), and let R(p,M,s) assign a score on the basis of this information. If the expert really believes that the uncertain quantities have (joint) distribution Q, then EQ(R(p,M,s)) is his expected score. Consider the M marginal distributions gotten from Q for each uncertain quantity, and let q = q1,...,qn denote the arithmetical average of these M marginal distributions. It is not difficult to show that q is also the expected relative frequency distribution under Q for the uncertain quantities. That is, qi is the expected relative frequency of outcome i under Q. Rule R(p,M,s) is a strictly proper scoring rule for average probabilities (or equivalently for expected relative frequencies) if

argmax(p) EQ(R(p,M,s)) = q.

In (R. Cooke, Experts in Uncertainty, Oxford U. 1991) a characterization of rules with this property is given. The score 2*M*I(s,p) is strictly proper in this sense. The global and item weights are "asymptotically strictly proper" in the following sense: Under suitable assumptions, for all p, if p <> q then for sufficiently large M (depending on p)

EQ(R(p,M,s)) > EQ(R(p,M,s)).


Coefficient of concordance

Let R(i,e) denote the rank of object i for expert i, where the rank is determined by the number of times e prefers i to some other object. Define

R(i) = SUM(e=1..E)R(i,e)

S = SUM(i=1..N)[(R(i) - (1/N)* SUM(j=1..N) R(j)]

The coefficient of concordance w is defined as

S

w = ------------- .

E^2*(N^3-N)/12

w attains its maximal value, 1, when there is complete agreement, and can be used to test the hypothesis that all agreements are due to chance (see Experts in Uncertainty, chapter 14).

The coefficient of concordance measures agreement of consistent preferences. The rank of an item, for an expert, is the number of times it is strictly preferred to some other item. Equality of preference is counted as one half of a preference. Hence If there are three objects, and the expert has a circular triad, all objects receive rank "1". If all experts have a circular triad, they receive rank "1" for all objects, and the coefficient of concordance is zero, even if they have the SAME circular triad. In the latter case the coefficient of agreement would be 1.


Item weights

Item weights are determined for each item separately, using (the global measure) calibration and the relative information for each item.


Chi-Squared probabilities table

The areas given across the top are the areas to the right of the critical value. To look up an area on the left, subtract it from one, and then look it up (ie: 0.05 on the left is 0.95 on the right)

df | 0.995 | 0.99 | 0.975 | 0.95 | 0.90 | 0.10 | 0.05 | 0.025 | 0.01 | 0.005
1 | --- | --- | 0.001 | 0.004 | 0.016 | 2.706 | 3.841 | 5.024 | 6.635 | 7.879
2 | 0.010 | 0.020 | 0.051 | 0.103 | 0.211 | 4.605 | 5.991 | 7.378 | 9.210 | 10.597
3 | 0.072 | 0.115 | 0.216 | 0.352 | 0.584 | 6.251 | 7.815 | 9.348 | 11.345 | 12.838
4 | 0.207 | 0.297 | 0.484 | 0.711 | 1.064 | 7.779 | 9.488 | 11.143 | 13.277 | 14.860
5 | 0.412 | 0.554 | 0.831 | 1.145 | 1.610 | 9.236 | 11.070 | 12.833 | 15.086 | 16.750
6 | 0.676 | 0.872 | 1.237 | 1.635 | 2.204 | 10.645 | 12.592 | 14.449 | 16.812 | 18.548
7 | 0.989 | 1.239 | 1.690 | 2.167 | 2.833 | 12.017 | 14.067 | 16.013 | 18.475 | 20.278
8 | 1.344 | 1.646 | 2.180 | 2.733 | 3.490 | 13.362 | 15.507 | 17.535 | 20.090 | 21.955
9 | 1.735 | 2.088 | 2.700 | 3.325 | 4.168 | 14.684 | 16.919 | 19.023 | 21.666 | 23.589
10 | 2.156 | 2.558 | 3.247 | 3.940 | 4.865 | 15.987 | 18.307 | 20.483 | 23.209 | 25.188
11 | 2.603 | 3.053 | 3.816 | 4.575 | 5.578 | 17.275 | 19.675 | 21.920 | 24.725 | 26.757
12 | 3.074 | 3.571 | 4.404 | 5.226 | 6.304 | 18.549 | 21.026 | 23.337 | 26.217 | 28.300
13 | 3.565 | 4.107 | 5.009 | 5.892 | 7.042 | 19.812 | 22.362 | 24.736 | 27.688 | 29.819
14 | 4.075 | 4.660 | 5.629 | 6.571 | 7.790 | 21.064 | 23.685 | 26.119 | 29.141 | 31.319
15 | 4.601 | 5.229 | 6.262 | 7.261 | 8.547 | 22.307 | 24.996 | 27.488 | 30.578 | 32.801
16 | 5.142 | 5.812 | 6.908 | 7.962 | 9.312 | 23.542 | 26.296 | 28.845 | 32.000 | 34.267
17 | 5.697 | 6.408 | 7.564 | 8.672 | 10.085 | 24.769 | 27.587 | 30.191 | 33.409 | 35.718
18 | 6.265 | 7.015 | 8.231 | 9.390 | 10.865 | 25.989 | 28.869 | 31.526 | 34.805 | 37.156
19 | 6.844 | 7.633 | 8.907 | 10.117 | 11.651 | 27.204 | 30.144 | 32.852 | 36.191 | 38.582
20 | 7.434 | 8.260 | 9.591 | 10.851 | 12.443 | 28.412 | 31.410 | 34.170 | 37.566 | 39.997
21 | 8.034 | 8.897 | 10.283 | 11.591 | 13.240 | 29.615 | 32.671 | 35.479 | 38.932 | 41.401
22 | 8.643 | 9.542 | 10.982 | 12.338 | 14.041 | 30.813 | 33.924 | 36.781 | 40.289 | 42.796
23 | 9.260 | 10.196 | 11.689 | 13.091 | 14.848 | 32.007 | 35.172 | 38.076 | 41.638 | 44.181
24 | 9.886 | 10.856 | 12.401 | 13.848 | 15.659 | 33.196 | 36.415 | 39.364 | 42.980 | 45.559
25 | 10.520 | 11.524 | 13.120 | 14.611 | 16.473 | 34.382 | 37.652 | 40.646 | 44.314 | 46.928
26 | 11.160 | 12.198 | 13.844 | 15.379 | 17.292 | 35.563 | 38.885 | 41.923 | 45.642 | 48.290
27 | 11.808 | 12.879 | 14.573 | 16.151 | 18.114 | 36.741 | 40.113 | 43.195 | 46.963 | 49.645
28 | 12.461 | 13.565 | 15.308 | 16.928 | 18.939 | 37.916 | 41.337 | 44.461 | 48.278 | 50.993
29 | 13.121 | 14.256 | 16.047 | 17.708 | 19.768 | 39.087 | 42.557 | 45.722 | 49.588 | 52.336
30 | 13.787 | 14.953 | 16.791 | 18.493 | 20.599 | 40.256 | 43.773 | 46.979 | 50.892 | 53.672
40 | 20.707 | 22.164 | 24.433 | 26.509 | 29.051 | 51.805 | 55.758 | 59.342 | 63.691 | 66.766
50 | 27.991 | 29.707 | 32.357 | 34.764 | 37.689 | 63.167 | 67.505 | 71.420 | 76.154 | 79.490
60 | 35.534 | 37.485 | 40.482 | 43.188 | 46.459 | 74.397 | 79.082 | 83.298 | 88.379 | 91.952
70 | 43.275 | 45.442 | 48.758 | 51.739 | 55.329 | 85.527 | 90.531 | 95.023 | 100.425 | 104.215
80 | 51.172 | 53.540 | 57.153 | 60.391 | 64.278 | 96.578 | 101.879 | 106.629 | 112.329 | 116.321
90 | 59.196 | 61.754 | 65.647 | 69.126 | 73.291 | 107.565 | 113.145 | 118.136 | 124.116 | 128.299
100 | 67.328 | 70.065 | 74.222 | 77.929 | 82.358 | 118.498 | 124.342 | 129.561 | 135.807 | 140.169


Calibration

Calibration measures the statistical likelihood of the hypothesis that the realizations are sampled independently from distributions agreeing with the expert's assessments. If s and p are the sample and theoretical distributions respectively for the the "inter-quantile interval" multinomial variable (see Bayesian Updates), and if I(s,p) denotes the relative information of s with respect to p), then calibration corresponds to the probability of seeing a deviation between s and p at least as great as I(s,p) under the above mentioned hypothesis. The larger this probability, the better the calibration. p is the same for all experts, as all experts assess the same quantiles, and s depends on the expert assessments. To indicate the dependence on expert e, we write s(e). The asymptotic value of this probability is

C(e) = 1 - Chisq_R ( 2*M*I(s(e),p) )

where Chisq_R is the cumulative Chi square distribution function with R degrees of freedom; R = the number of quantiles, M is the number of seed variables and e is the expert in question.


Thurstone linear

In this case it is assumed that the true values Pi for object i, i = 1,...,N, are related to the means from either model B or C by some positive affine transformation:

Pi = a*mi + b; a > 0.

If the true values are known for two objects, (reference values) the above equations may be solved for a and b, and the values for the remaining objects may be computed.


Confidence intervals

Confidence intervals are simulated for the Thurstone C linear and the NEL models. The procedure is roughly as follows:

1. The model is solved using the expert data and reference values. This yields values for the model parameters mi (Thurstone c) or li (NEL).

2. Expert responses are simulated by appropriately sampling distributions with the model parameter values emerging from step 1.

3. The model is solved again, using the expert preference data emerging from step 2, and the initial reference values. The results are stored.

4. Steps 2 and 3 are repeated 50 times for each model and 90% central confidence bands are extracted and displayed.


Robustness (experts)

Seed experts are excluded from the analysis, one at a time and the resulting DMs are computed and compared, using the current parameter values under "RUN". The total relative information with respect to the background measure, the calibration and the total relative information with respect to the original DM are tallied.


Calibration power

The power of a statistical test is its ability to distinguish between rival hypotheses, and increases with the number of independent samples. Calibration power may be chosen from the interval [0.1, 1.0], and determines the effective number of samples. Choosing 50% power means reducing the resolution of the significance test to that of a test with half the number of samples.

Instead of calculating experts calibration with the formula (see Calibration)

the following formula is used in calculations:

where


Table of content

Table of contents

INTRODUCTION

Welcome

References

Chi-Squared probabilities table

GETTING STARTED

Preliminaries

Entering Data

Calculating

Display Results

GLOSSARY FOR EXCALIBUR

Calculate

Bayesian updates

Calibration power

Discrepancy

DM

Gobal weights

Intrinsic range

Item weights

Optimized DM

Robustness (experts)

Robustness (items)

Significance level

User weights

Display results

Calibration

Normalized weight NO DM

Normalized weight with DM

Relative information

Strictly proper scoring rules

Unnormalized weight

Items

Quantile

Quantile point

Scale log/uni

Weights


(Untitled)

{ewl RoboEx32.dll, WinHelp2000, }


Gobal weights

Global weights are determined by global measures of performance on seed variables, namely, calibration and average (over seed variables) relative information. When the significance level is set equal to zero, an expert's global weight is proportional to the product of calibration and average relative information over seed variables. For each expert, global weights are the same for all items.


Thurstone models

It is assumed that a subject comparing two objects i and j effectively samples two normal random variables Xi and Xj with means and standard deviations mi,si mj,sj. Object i is preferred to j if and only if Xi - Xj > 0. It is further assumed that all subjects sample the same normal variables. The variable Xij = Xi - Xj is normal with mean mij and standard deviation sij:

mij = mi - mj

sij = (si + sj - 2rij*si*sj)

where rij is the correlation between Xi and Xj. The percentage %(ij) of experts preferring i to j estimates the probability that Xij is positive, or equivalently, that (Xij- mij)/sij > -mij/sij. Letting F denote the cumulative standard normal distribution function, and let F^(-1) denote its inverse. We derive Thurstone's law of comparative judgments:

F^(-1)(%(ij)) ~ mij/sij = (mi - mj)/sij.

Note that the law of comparative judgments is invariant under positive affine transformations of all variables. That is, if for each i, Xi is replaced by a positive affine transform Xi' = aXi + b, a > 0, then mij'/sij' = mij/sij. This means that paired comparison data can determine the values of the mi only up to a positive affine transformation, and the values of si only up to a linear transformation.

Considered as a system of equations, the law of comparative judgments contains more unknowns than knowns, and hence cannot be solved. However, under various assumptions it is possible to determine the means mi up to a positive affine transformation, i.e. up to a choice of zero and unit.


DM

Decision Maker:

The DM's distributions are determined as the weighted combinations of the experts' assessments. If [image] is the cumulative distribution function of expert i for a given item, and NE is the number of experts, then:

where [image] are the global, item weights or user weights.


Entering Data

Entering Items:

ENTER creates a new item when the cursor is at the end of the table;

INSERT inserts a blank item after the current one, and ALT-INSERT – before the current one.

DEL deletes an item when in the row mode.

The standard Cut/Paste/Clear functions are available in the EDIT menu when in the row mode, as well as the CTRL-Z (Undo).

Scale is either LOG or UNI. Scale must be edited from this menu.

Scale of all variables can be changed from the EDIT menu.

Numerical format can also be changed from the EDIT menu.

Ctrl-C copies the active window to the clipboard.

Entering Experts:

ENTER creates a new expert when the cursor is at the end of the table;

INSERT inserts a blank expert after the current one, and ALT-INSERT – before the current one.

DEL deletes an item when in the row mode.

The standard Cut/Paste/Clear functions are available in the EDIT menu when in the row mode, as well as CTRL-Z (Undo).

DOUBLE CLICK activates the assessments window for the expert under the cursor.

CTRL-C copies the active window to the clipboard.

Entering Assessments:

ENTER, CURSOR ARROWS and TAB: accept the data entry from the cell mode.

RIGHT CLICK: parametric assessments menu.

If assessments are already available, an estimate of appropriate parameter values is shown default when a distribution type is selected. These parameter values may be changed, and the assessment is adjusted accordingly. If no assessment has been entered, switching to the row mode and RIGHT CLICK enables parametric assessment.

The row mode is not available from Assessments.

If a variable is on LOG scale, assessments must be greater than zero.


Scale log/uni

Variables measured on a uniform scale (UNI) are uniformly distributed between quantiles, and relative information is measured with respect to the uniform distribution (suitably truncated).

Variables measured on a loguniform scale (LOG) are loguniformly distributed between quantiles, and relative information is measured with respect to the loguniform distribution (suitably truncated).


(Untitled)

Welcome

EXCALIBUR (acronym for EXpert CALIBRation) is a Windows program that allows parametric and quantile input from experts for continuous uncertain quantities and combines these according to the methods described in R. M. Cooke "Experts in Uncertainty", Oxford University Press 1991. In particular user-weights, equal weights and performance based weights are supported. Robustness analysis shows how sensitive the results are to choice of expert and choice of calibration variables. Discrepancy analysis shows how the experts differ from a Decision Maker. Output is compatible with modern text processors and spreadsheets. There is also a paired comparison module (WCOMPAIR) which implements the Thurstone and Bradley-Terry models.

The light version of EXCALIBUR may be downloaded free. This version is restricted to 5 experts and 15 items. A user manual is provided with the program (available upon request).
EXCALIBUR sales are not intended to cover development costs and generate profit. Rather it is token-ware: We ask a token price of $100 for package plus user manual to cover shipping/handling costs, and to demonstrate to administrations that scientific research, though not a commercial activity, is not void of commercial value.

This new all-in installation package includes also a help file and the WCompair utility for paired comparisons. IMPORTANT: If you have already a previous version of Excalibur installed, please uninstall it first, before installing the new one. For more details on this release, please refer to the Readme.txt located in the main Excalibur directory.
The Light version of Excalibur has two limitations, comparing to the Pro one: it supports only max 20 experts and only max 100 input variables (the Pro supports up to 1000 of experts and items, WCompair Pro up to 150), otherwise they are identical. To order site licenses for Excalibur Pro, please visit http://ssor.twi.tudelft.nl/~risk.

This version of the program has been compiled on April 5th, 2004.


Normalized weight with DM

These weights result from normalizing the experts and the DM's unnormalized weights.


Weights

Weights are used to determine the Decision Maker's distribution, as a normalized weighted linear combination of the experts' distributions (see DM). In EXCALIBUR the user may choose one of four weighting schemes, namely global weights, equal weights, item weights and user weights.


Significance level

The significance level determines the calibration threshold value (see Calibration). Calibration scores greater or equal to the significance level correspond to non-rejected statistical hypotheses. The unnormalized global and item weights are asymptotically strictly proper scoring rules (see proper scoring rule or R.M.Cooke "Experts in Uncertainty", Oxford University Press, 1991, chapter 9) only if combined with significance testing at a non-zero significance level. The significance testing entails that the weights become zero whenever the calibration score is strictly less than the significance level. The theory of strictly proper scoring rules does not determine what the significance level should be, this is determined by optimization.


Coefficient of agreement

Let E be the number of experts, N the number of object is to be compared, and a(ij) the number of experts who prefer object i to j. Define "A CHOOSE B" as the binomial coefficient A!/(B!(A-B)!), and define

Delta = SUM(i<>j) (a(ij) CHOOSE 2)

The coefficient of agreement u is defined as

2*Delta

u = --------------------------- - 1.

(E CHOOSE 2) * (N CHOOSE 2)

u attains its maximal value, 1, when there is complete agreement, its minimal value, 1/(E-1) is obtained when half of the experts prefer i to j, for all i,j. u can be used to test the hypothesis that all agreements are due to chance (see Experts in Uncertainty chapter 14).

The coefficient of agreement measures agreement irrespective of intrasitivities, that is, experts can have many intrasitivities and yet agree completely. Equality of preference, or empty preference are both interpreted as non-preference. Hence if all experts have equal or no preference between all objects, the coefficient of agreement is -1.


Normalized weight NO DM

These are the weights used in determining the DM. If equal or user weights are used, then these are shown, if global weights are used, then the normalized global weights are shown. If item weights are used, then this column is blank, since the weights used for the DM vary from item to item.