How do Report Cards help?
As outlined in the section on Transparency and the right to information, the concept of transparency is often very difficult to define and to accurately measure and just as there is no commonly agreed definition of "transparency", there is no consensus on how it should be measured.
But since transparency is seen as a mechanism for promoting accountability, then one key way of measuring transparency is to measure the amount (and scope, accessibility, quality, reliability, accuracy and timeliness) of information disclosed and/or made publicly available.
Global Witness, one of the five partner NGOs working on Making the Forest Sector Transparent, has, for example, previously used quantity, quality and credibility of information as a measurement of transparency in its Independent Forest Monitoring
How report cards can help
To take this measurement process one step further, the partners of this project, through a participatory process, have developed a transparency report card as an assessment tool.
A report card is basically an assessment tool. Whilst report cards are popular in other sectors, the use of a transparency report card for the forest sector is innovative.
Broadly speaking, there are two main models: (i) Scoring and Ranking or (ii) Descriptive.
Scoring and ranking report cards are quantitative in nature. Scores are awarded according to a standardised scale (anything from simple binary scoring -- 1 or 0, for example -- to different categories from which to choose, such as green yellow or red.) The scores obtained are usually aggregated into a single index. Since any scoring implies some sort of weighting, rankings can be established and broad comparisons can be made between units of analysis.(Image shows an example of ranking from Open Budget Initiative).
Descriptive report cards are more qualitative in nature. They rely on the compilation and analysis of a series of different criteria and indicators. Descriptive report cards allow for a greater level of analysis on a case-by-case basis.
The advantage of report cards
Some of the key advantages of report cards as assessment tools include:
- They typically contain "yes/no" questions, helping to increase objectivity;
- Data can be gathered and compiled quickly;
- Use of a standardised format and sets of assessment indicators which are easily replicated makes report cards powerful tools for making comparisons over time and/or across units of analysis;
- The combination of objective "yes/no" data and more discursive analysis helps to identify priorities for follow-up work.
Report cards as a tool for monitoring forest sector transparency
By way of example, the report card developed for Making the Forest Sector Transparent gathers data by asking 70 questions split across 15 agreed-upon "themes" -- each of which is a measure of accountability. Data is collected in each country on a yearly basis.
Under each of these general themes, there is a sub-set of related questions: for example, an overall question might be "Is the permit allocation process transparent?" and respondents can answer "yes" or "no" to this question. But the answers to a series of more probing sub-questions helps us to understand how they have arrived at this yes or no answer and help guide any advocacy work. For example, respondent are then asked to clarify "Do permits exist for all users/services?" and "Is it clear who decides if/when to allocate permits?" and so on.
Key considerations when developing a report card
Report cards are only useful as an assessment tool if you very clear about what it is you want to assess and if the information you collect is relevant and reliable. It is therefore crucial that, during the design stage, you:
- Know enough about the purpose for which you want to use it and you need to define what you are going to assess;
- Assess how many questions to include, taking into account: the desired coverage; the desired level of specificity; issues of feasibility and availability of information; and the intended audience. A good mantra is "don't ask a question if you are not going to make use of the answer";
- Avoid obtaining subjective or non-comparable answers, either by (i) setting very specific "yes/no" questions or (ii) establishing clear assessment criteria (for example, providing further, more specific options for each "yes/no" answer in the form of a,b,c,d or e..);
- Be able to clearly describe and explain the rationale behind the criteria and indicators used; this will help to give credence to the research as well as help explain the findings.
