Mizzaro's framework for relevance

2. Components

An important contrast here is that of topic vs. context. Stefano emphasises subject matter vs. broader domain background; I would emphasise information (e.g. a fact) needed vs. meta-information that may modify the goal (information need) (e.g. there are many papers with almost the same content so no one citation is the best here).

3. Time

One interesting example is a dictionary that would let you understand a document in another language. Given the task, they are only relevant in combination, so a) their relevance depends on each other, b) if you you retrieve one at a time, then the relevance of the first depends upon the future (whether you retrieve the second).

4. Information resources

Steve Draper's notes on possible problems

1. Binary or scalar relevance judgements. (But see [7] below.]
   In traditional IR, human judges are asked to make binary judgements of a document's relevance, while the software is required to assign a scalar or at least ordinal score. Comparing the two (apples and oranges) is obviously a mistake. We could require the software to make binary judgements, but that would mean offered lists of documents were no longer sorted by estimated relevance, which is actually clearly one of the biggest benefits of IR as opposed to database retrieval. So it seems clear that we should ask experts to make scalar not binary judgements of relevance in order to test IR software properly.

   This is the subject of Fabio's session. It seems clearly right and necessary. Stefano's framework does not really help here: it seems more likely to lead to eliciting a large set of binary judgements (one for each type of relevance), rather than the ordinal score required.

   Now Stefano might respond that even though he talks in terms of binary judgements, we could easily take a scalar measure for each of his dimensions. But there is a further crucial point here. A single ordinal score is almost certainly what is truly necessary in the software, because only that allows the user to scan a list in order of merit. Even if it is true that relevance is "really" multi-dimensional, software will not be useful unless it can produce one-dimensional ordered lists for users because the user's task is a decision task: choosing a single document to examine next. This argument is essentially similar to those about cost-benefit calculations and whether all kinds of benefits (human life, works of art, etc.) can all be reduced to the single measure of money. The argument about decision tasks says yes: a choice must be made, so everything must be compared on a single ordinal dimension.

2. Information needs should be distinguished by representation, not cause
   Stefano's approach to the information need dimension is to define the distinctions in terms of the representation of the need. This is an attractive approach, as it seems easy to relate it to operational measures: if the need is expressed in natural language then it is a request etc. This is better than, say, an attempt to make distinctions based on time sequence e.g. before the user expresses it, after it ...: because a need may well begin life as a written request to the user from their manager, and the user then adds background knowledge to create a PIN by working "back" from the request. Thus Stefano's definition of information need type by representation is a good move because a) it is independent of time (as I showed, the EIN may come before or after the PIN in historical sequence; and that is one reason why he has time as a separate dimension); b) it is independent of cause (the EIN may come from the PIN, or vice versa).

3. External, socially specified information needs.
   Jane Reid points out that many information needs come to a user from another person e.g. your boss orders you to do some task, which will require you to do some IR. The framework allows for this in one way well (distinguishing between what is spoken or written about the need, from what is implicit in background knowledge). However it does not explicitly allow for these two (or more) agents. Yet this is an important feature of many real life IR tasks, and we may need a framework that describes this. For instance, software that supports the interaction about the IR task between humans may be more useful in many cases that software that does not, but we could not describe the difference using only Stefano's framework.

4. Information needs: are they ordered?
   Stefano's approach is to have an ordered dimension of information need. But is there really an order there, or only an unordered set? As argued above, at first it seems like the order is that of time and cause (RIN comes first, and then the PIN is made from it, etc.); but that is often not what happens e.g. the Request may come first and the PIN be created from it.

   Another possible order is that of completeness and detail: the amount of information in each representation. The RIN seems the whole thing, and the PIN a subset (the part the user consciously realises), and then the request is a subset of that (the part the user knows how and decides to express). But this may not always be the case. A query may be more precise than the request, and only the requirement of having to formalise the query may make the user calculate exactly what they need. Or a user may believe (the PIN) they need complete information for a task, when in fact (the RIN) they only need approximate information: e.g. they request a photo to recognise someone, but in fact there is only one person in the room they do not already know so recognition is easy; or they request a full bibliographic citation, but in fact just the journal and author is enough because there is only one paper by an author of that name.

   So there are apparent orderings by time, causality, and amount (completeness) of information, but none of these is valid and general. The information need dimension must be unordered.

   [Jane Reid's comment] The discrepancy between these different information need elements is important, not the ordering. To me, the vital point of time is where the PIN (against which RJs are made by the user) most closely resembles the RIN, i.e. the point at which the user has the best, closest and clearest idea of his "real" information need. It is for this reason that I am using post-feedback RJs in my PhD work.

5. It follows that a complete description of a task's information need would require the following kinds of description:
   • The set of types of information need in terms of type of representation
   • The time sequence in which various representations appeared
   • The causal sequence (actually, partial order) of these
   • The sequence or set of human agents involved in formulating some of these.

   ---

   Measuring relevance

   The following further comments arose from a conversation with Jane Reid on 6 Aug 1998.

   We need to measure relevance (e.g. in creating test collections) in order to test software. And in connection with that, we are interested in the question of the degree of consensus between judges.

   This involves thinking of relevance judgements (RJs) as actions (not abstract, eternal value quantities), and of how they can be measured. The usual way is to ask experts to give a judgement. In contrast, in current IR systems the user typically makes a decision e.g. about whether to open a document within one or two seconds: this cannot be sensibly considered as reflecting the user's best evaluation of a document's relevance. Research on relevance judgements must equally be beware of this: asking people about relevance is not necessarily much connected to what their behaviour would be when using IR software.

   So note that the main reason for studying RJs is to support user actions better while they are using IR software. We want ultimately to support their making optimal actions. Relevance, however defined, is only part of this. Note too that experiments on RJs are a kind of action, but usually a different kind of action, so we need to be aware of these issues in order to relate experimental findings with user behaviour during IR, rather than naively equating them.

6. Time to make the relevance judgement: do we ask them to make the judgement in (say) 2 seconds maximum (i.e. our estimate of the time taken to make most user decisions in current IR), or do we ask them to reflect and discuss and come up with a final judgement. Of course, now I ask this, we probably want to run studies where we do both, and report on whether there is a difference in the judgements made.

7. RJs may be
   • binary (the traditional way of collecting them in test collections); or
   • scalar (e.g. scored as numbers between zero and one i.e. this are absolute values); or
   • ordinal (the judge ranks the set of documents in a full ordering i.e. relative relevance); or
   • just the single most relevant is selected. Note that in current IR software, the decision users actually have to take is just to pick one action from a set of (typically) 12: 10 ranked documents, or choose to look at the next 10 surrogates, or choose to abandon this query and/or task. None of the four alternative ways of measuring relevance just listed actually correspond to the user's decision task. A fifth alternative would be:
   • the actual 12-way choice. This seems like the binary RJ, but unlike the binary RJ we would expect it to depend in part on whether there are more surrogates to view, whether the user could think of different queries to try, etc.
   • Another measure that perhaps should be collected is the degree of consensus about a judgement. In experiments, we would like to collect this to teach us more about the nature of RJs. But in software, such a number could be useful too: it would be like having a sensitivity or confidence score associated with the RJ score, and this could be used to display to the user, to decide on whether and where to threshold lists if this is done, and so on.

8. Group relevance judgements. An interesting way to study RJs, though it does raise still more issues, is to have a set of users or experts discuss an RJ to see if this increases consensus. After all, if there is a lot of variation in RJ values this could be because there is a lot of genuine disagreement, or because some users missed some point but would quickly agree on the "real" RJ value if this is pointed out. One possible format for studying this might be to give an RJ problem to a group of people; ask each person separately to give an RJ instantly (within a 2 second limit); then to think more carefully until they are happy with an opinion and to record that; then to discuss it as a group and agree a single group RJ; then to give again a private RJ. This design can show the effect of group discussion (if any) on private judgements.

9. Look at IR and relevance in the wild i.e. while a user is doing IR, and measure both their behaviour (what documents they select e.g. to open) AND what they say about its relevance (e.g. as soon as they select one, ask them how relevant they think it is i.e. ask why they selected it). There may well be cases where users select a document while saying it is not relevant. (Possible cases: a retrieved document says "free holiday for the first to read this" even though that is not related to the query; i.e. money, sex, etc. are likely to motivate the user independently of their IR task; or a document that says "Today's tips for improving your retrieval methods with this collection"; i.e. the user may decide that the document will help them indirectly, and so describe this as "irrelevant" even though it will help with the information need.) There are broadly two possible cases, if such divergences are in fact observed. Firstly, can the relevance framework cover the triggering of new information needs; secondly, users may use the word "relevance" when asked in a narrower sense than is needed for a comprehensive framework like Mizzaro's, which must affect how we do research on relevance.

Conclusion

Measuring relevance

Relevance framework

Furthermore [Jane Reid's comment] the context is defined as "information about the subject domain used during the activity". I don't recall whether this is exactly how Stefano himself described it, but, if so, this seems to me rather a narrow interpretation of context. There will be, after all, contextual factors which have little or nothing to do with the domain which may well influence RJs. These include other indirect domain information the user might bring to bear, e.g. "my supervisor doesn't think very much of this researcher, so any papers by him are worth little to me", or completely independent factors, e.g. what level of enthusiasm or motivation the user has on a particular day. I can think of lots of different examples in these areas. In other words, I think either:

• the definition of context needs to be widened to take this into account or
• there is really a SET of contexts, rather than one simple one, e.g. a domain context, a task context, a topic context...

The issues of speed of making a judgement and whether discussion modifies judgements is important for measuring relevance, but may not affect the framework.

Points about devising demonstrations

Demonstrations are mainly single examples. A single example cannot prove a point about all possible cases, but they can prove some things. A single counterexample disproves a theory (or other universal statement); a single example proves that a proposition is possible in the world. A single example that strikes everyone as absolutely typical can prove that an issue is important (not just applying to special and unusual cases).

So for each phenomenon (e.g. how much consensus is there in relevance judgements?), we might look for 3 demo. cases: One extreme (a case where everyone agrees), the other extreme (a case where everyone disagrees), and a prototypical case (a case that seems extremely common, and if there is some disagreement about relevance, then lack of consensus must be a significant issue in general).

Cases to look for?

• A porn or money item inserted in a retrieval list, to show people selecting a document to look at which they say is not relevant.
• A bad or misleading surrogate to demonstrate a case where people's fast (2 second) relevance judgement is different from their slower, more careful judgement.
• A case to show up differences between binary, ordering, and scalar RJs e.g. a query "door on left of window" and picture with the door on right of window. (In this case, the picture is absolutely irrelevant and might be scored zero, yet will probably be ranked as more worth a second look than pictures with no doors and no windows.)

Points to debate

• Do we need binary relevance judgements, or scalar (quantitative) ones?
• Or, further, which of: binary, scalar, ordinal, single most relevant document, single most relevant action, degree of consensus?
• Must we have a single relevance judgement, even though relevance is really multi-dimensional?
• Is the information need dimension unordered or ordered?

(Back up to notes on MIRA Dublin workshop     Back up to Steve Draper's home page)