What to include in a 'critical evaluation' of a research report

It is common at British universities to ask undergraduate students to write a critical evaluation of a research report from a journal in your subject area.  This type of assignment asks you to read critically.

If this is one of your assignments, then here's a list of questions you need to ask yourself while reading and to address in your written evaluation.  Of course, these questions apply equally to your own work; in other words, use these to evaluate critically any write-up of your own primary research (e.g. your dissertation) before submitting it to your tutor.

The journal

  • Is it peer-reviewed? In other words, are articles read/reviewed/commented on by academic peers and revised before accepted for publication? 

The title

  • Is it clear and concise? Do you know what this report is going to be about?

The author/s

  • What do you know about the authors? Did they receive any funding for the project? If so, from whom/what organization? Is there any potential conflict of interest or bias?

The abstract

  • Does it summarize the main points of the study adequately and accurately?

 The introduction

a)     The problem:

  • Is the problem or purpose of the study clearly stated?
  • Is the problem and/or key terminology associated with it defined?

b)     The question:

  • Is it/are they stated clearly and concisely?
  • Does it/do they follow logically from the problem?
  • Are they worth answering? Better yet, are they answerable?

c)     The need:

  • Is the significance of the research stated, i.e. does the author carve a niche for his/her own study?  Is it making a contribution to our current knowledge?

d)     The aims:

  • Are they stated clearly and concisely? Are they logically related to the original question?

The literature review:

  • Is the background information sufficient? (Does it leave you asking more questions?)
  • Does the author appear to know his/her subject?
  • Is the author critical (positive & negative) of related research? Or has he/she patched together quotes/paraphrases which support her/his own position without considering the counter-arguments?
  • Are specific theories/frameworks used in order to put the study/findings into context?
  • Does the theory seem relevant?

 Method:

a)     Design

  • Is it described in detail?
  • Does it follow logically from the original problem?
  • Is it narrative (words), numerical (numbers) or both (mixed-methods)?
  • Was a pilot study completed, e.g. were the methods tried out first on someone/non-participant equivalents? Were changes made? What and why?

b)     Ethical considerations

  • Does the author adhere to specific ethical guidelines? If yes, which one/s?
  • How does the author make his/her research ethical, e.g. was consent granted? Are pseudonyms used? Were participants informed that they could: ask questions; withdraw from the study; gain access to the findings, etc…?

c)     Samples and participants

  • Who was chosen? How were people selected?
  • How many people were selected? Why this many?
  • Was a specific size of sample chosen for a reason (e.g. statistical purposes)?
  • Is the sample representative? Does it need to be, e.g. does the author intend to make generalisations?

d)     Data collection

  • What methods are used to collect data (e.g. questionnaires, interviews, focus groups)?
  • Are the methods described adequately?
  • Could you replicate this piece of research from the description?
  • Are the questionnaires, interview schedules etc. included in the text or appendix?

e)     Analysis

  • Is the method of analysis understandable?
  • Are reasons given for the type of analysis? Do these adequately justify the chosen method?
  • If numerical analysis, what kind of statistical tests were used? Does the author explain these well enough?
  • If narrative analysis, was there evidence of reading/re-reading/coding/categorizing the data?
  • Was it done by one researcher or more (e.g. inter-rater reliability)?

Results

  • Are results presented in a clear and coherent way so that you can interpret them and come to your own conclusions?
  • Are raw data supplied or only proportions, percentages, etc. after manipulation? Are actual quotes given? Are numbers given before turned into proportions/percentages?
  • Are charts, tables and other graphic representations labelled and explained?
  • If results are based on responses to a questionnaire what is the response rate?
  • Are statistical results included? Are they meaningful?

Discussion

  • Are the results interpreted in relation to the original question?
  • Are the original questions answered?
  • Does the author discuss any weaknesses in the methods and factors which may have affected validity (i.e. accuracy of assessment - the study measures what it sets out to measure) and reliability (consistency of assessment - the study can be replicated by someone else)?
  • Do findings link back to/make comparisons with previous studies reported in the literature review?
  • Are implications discussed?

Conclusions

  • Do you reach the same conclusions based on your interpretation?
  • Do they relate logically to the results?

Recommendations

  • Are there recommendations for practice or for further research?
  • Could you attempt to implement them (should you)?

References

  • Is the length of the list more impressive than its quality?
  • Some work is seminal and important to the field and may have older (old) publication dates.  This is fine, but there should also be up-to-date sources. So, look for a mix.
  • Are there any references not included which you think should be?