Literature reviews are an integral part of research, since they provide a summary of current research on a topic. Literature reviews might have any research question as their theme. This paper presents an evaluation of the literature review methodologies, it covers the narrative, systematic and meta-analysis literature reviews. It also highlights some of the most important benefits and restrictions of each methodology.
There are plenty of methodologies and guides on how to write a good literature review throughout science principles, illustrating a variety of strengths and weaknesses. This paper documents a literature review to explore the ability of structuring a “good” scientific literature review, as well as the most valuable methodology in the field.
A consensus exists in literature in regards of the most scientifically approved methodology, which, according to Marshall, Goldbart, Pickstone & Roulstone, is the systematic literature review (2011, p. 261). However, the literature suggests three general methodologies of authoring a literature review, each of them has more than one subcategory and variations. (Haneline, 2007 p. 19) Recently, the most popular method of conducting a literature review, is considered from many the “rapid review”, which is a variation of a systematic literature review. This method shines under strict time schedule circumstances, since according to its prodigies “the word ‘rapid’ indicates that it will be carried out quickly”. (Harker & Kleijnen, 2012 p. 398)
There is a plethora of literature review definitions, ranging from simple but useful definitions, such as the one provided by Kumar, stating that the literature review is an indispensable and valuable stage of research; to very complicated definitions which can be arguably hard to comprehend. Eventually, the definition that applies to this paper is a combination of the definitions specified by Marshall G. (2010), as well as Smallbone & Quinton (2011); both papers conclude that a literature review can be defined as “a systematic method for identifying, evaluating and interpreting work produced by researchers, scholars and practitioners”. Kowalczyk & Truluck go a step further, arguing that “a good literature review requires the author to provide critical analysis of the literature”. (2013) There are, however, definitions that are considered out of scope of this research, such as the one proposed by Hart “The purpose of the research is to contribute in some way to our understanding of the world” (1999, p. 12), which would fit better a medical based paper, as a high level definition.
The research questions addressed in this literature review are: (a) what defines a good literature review? ; And (b) what are the different methodologies available, for a successfully undertaken literature review?
The analysis and finding are documented as follows. Section 2 describes the research method for selection and analysis of the books and papers. Section 3 presents the review of the literature process, followed by the methodology choices in Section 4. Section 5 concludes the paper.
2. Research methodology
The review of the literature relating to the quality of a literature review with a focus on science was undertaken in December 2015, using Discovery along with ResearchGate and Google Scholar search tools and databases for completeness.
2.1. Selection of literature
The initial research which resulted in 5 publications after omitting the duplicate papers, involved the search “what is literature review” in Discovery, narrowing the 134 initial results to English academic journals, having as subject the key words Literature review and being published later than 2005. The complete search algorithm was:
“What is literature review + methodology + Research + Evidence-Based + Literature Review Methods, during past 10 years, only academic papers or e-books or books”
A total of 27 papers, 2 e-books and 9 books were selected for detailed review, after removing the 10 duplicate papers; from those 15 papers were discarded due to failing to comply with the inclusion criteria as stated below:
The inclusion criteria for each item were that the paper of the book:
i. Has been commercially published, republished, or reprinted in the previous 10 years; is still available; and can be accessed or purchased by the library of the University of Portsmouth;
ii. It has been published or translated in English or Greek;
iii. Provide an implication that a literature review methodology and/or definition is described in the paper or book by either the title or the abstract.
iv. Contains a comprehensive list of references;
v. Is substantial in size and the author(s) accreditations are relatively straightforward and easy to find.
A list of the above papers for better comprehension of the inclusion criteria, can be found in the Appendix A. Assuredly, several secondary searches were initiated for akin search topics and key cited articles determined in the initial research.
3. Reviewing the literature
Reviewing the literature is an ongoing process, throughout the research lifecycle. Ridley correlates that continuing process with a “recursive literature review” (2012 p. 19). An accord exists in literature that the first step into writing a good literature review is to define the research question, to which the review produced should be relevant. (Kumar, 2014 p. 60)
The review of the literature depends on four key factors, namely methods to search the literature, quality of literature being searched, methodology of organizing the research results and referencing. It is typical amongst authors to provide a few advices and common criteria, of what they regard as a good literature review. (Mallidou, 2014 p. 32 and Kumar, 2014 p. 60) However, according to Kowalczyk & Truluck, a good literature review is not just a collection of literature numbered and summarized, but a critical evaluation of the collected literature (2013 p. 219). An impressive fact, is that both Marshall (2010) as well as Wong, Greenhalgh, Westhorp, Buckingham & Pawson, (2013) put a lot of effort to explain how to critically evaluate the resourced material, but they do not mention acceptable sources and methodologies for collecting those. Actually, there are only a few authors that mention the difference between the types of literature and what can be considered a “good” literature source; the vast majority of whom, they also support a variety of literature review methodologies and do not focus only on systematic literature reviews and their variations.
3.1. Inclusion Criteria
The main sources of identifying literature are books, journals, conference papers and the Internet. (Kumar, 2014 p.60) Importantly, the main criteria for appraising the collected literature focusing on the above sources are described by Mallidou (2014, p.33) and they can be encapsulated as the use of researcher’s experience and judgement, which can lead to bias, is unavoidable; therefore the application of a generic standardised instrument or algorithm would minimise the bias (Ridley, 2012) and (Gough, Oliver & Thomas, 2012).
The element of bias as well as the personal experience and judgements lead to the final and arguably the most important fact of a literature review search, which is the inclusion criteria. Literature reviews that have numbered and clearly stated inclusion criteria are considered to be more accurate by the majority of the readers. (Marshall, 2010 p. 20) Although this can be considered true for some literature reviews, McCrae, Blackstock & Purssell support that plethora of reviews specify exclusion criteria instead of inclusion, (2015, p. 1272) argument that is also supported by Wright & McSherry (2013, p. 1362). The incorporation of the exclusion criteria instead of the inclusion, is considered “more useful by cause of the eligibility being framed negatively, instead of a positive specification with clearly stated conditions”. (McCrae, Blackstock & Purssell, 2015 p. 1272) An interesting observation in regard to which type of criteria should a writer using is able to be made here, since the majority of authors do not differentiate between “exclusion or inclusion” criteria, they however, focus towards clarity and simple definition of one of those criteria categories. (Flemming & Briggs, 2007 p. 97)
3.2. Critical thinking in literature review
All the literature review methodologies presented in Section 4, require the literature review to be critical. However, there is often a confusion in literature between critical thinking and criticism. Critiquing the literature is considered one of the hardest parts of writing a literature review, due to the fact that they depend upon deeper thought and making connections unconcerned of the type of methodology the author used. (Marshall, 2010 p. 22)
It is advised that the researcher should research the credentials of the sources, in a tactful but not destructive manner, supported by well-defined evidence. The literature review does not target the author’s personality, but uses a sceptical attitude based on reasoning and common sense to criticise the findings. (Thody, 2006 p.98) Furthermore, it is also advised that the researcher should evaluate the author’s arguments, assumptions as well as findings and conclusions based on the provided evidence only. (Ridley, 2012 p. 142)
4. Methodological choices of a literature review
Authors such as Wong, Greenhalgh, Westhorp, Buckingham & Pawson, (2013 p. 1006), Kowalczyk & Truluck (2013 p.219) and Marshall (2010 p. 20) are openly in favour of a systematic literature review as the methodology of choice. There are, however authors that distinguish the literature review, methodology choice further and they state that depending on the science, time frame, resources and the data analysis method the researcher is applying, the literature review methodology should vary. (Walsh & Downe, 2005 p. 204-205) The latter statement is likewise supported by Mallidou, who reasons that the literature review, methodology of selection is also depending on the literature search the researcher is performing. (2014 p. 36-38) This review analysed papers and books correlating the different literature review methodologies, identifying that it is commonly stated amongst the literature that there are three “main” literature review methodologies, or as a Haneline addresses them literature review designs. (2007, p. 19)
Types of literature review methodologies:
i. Narrative review
ii. Systematic review
Each of the above methodologies, have their own strengths and weaknesses, their “preferred” applications as well as their supporters. Nevertheless, it is admitted by every paper and book this review analysed, that there are immense gaps on the methodology, definitions and it can be very complex to identify which literature review methodology suffices the most, in which research scenario. Watt, Cameron, Sturm, Lathlean, Babidge, Blamey, Facey, Hailey, Norderhaug & Maddern (2008)
4.1. Narrative literature review
A narrative literature review often covers a subject broadly, performing a selective review of the literature, without following any systematic methods to locate and evaluate articles, fact that makes it almost impossible to duplicate. (Heinlein 2007 p. 19). This definition leads to the designation of four crucial drawbacks that narrative literature reviews have in comparison to the other two methodologies. Firstly, they are prone to bias at the stage of researching and evaluating the resources and as a result, they may produce biased results. Furthermore, they might be one sided, supporting the author’s point of view. Moreover, the method of producing their results is usually unclear; hence it is common to count the number of papers in favour of one argument in comparison with the number that supports the opposite argument. (Haneline, 2007 p. 20) Finally, it is considered confusing by the literature to differentiate between a narrative critical and a systematic review, due to the absence of use of rigorous methods to identify, assess and combine articles, to the point that McAlister & Clark consider the validity of those narrative reviews frail. (1999, p. 949) There are also, authors that are openly against a stand-alone narrative literature review, such as Vogt, Gardner & Haeffele, who define a narrative literature review as an introductory review, who’s a systematic or a meta-analysis will follow. (2012, p. 90) This perspective is also supported by Ridley stating that a narrative review should be incorporated in a systematic literature review. (2012, p. 195)
Despite the constraints of narrative reviews, they can be truly valuable to researchers. They are frequently mentioned as narrative, critical literature reviews. Although, completing a narrative critical literature review can be challenging, especially due to the different expectations that each scientific discipline has, every discipline requires a critical review, which has to be of good quality, original and perceptive. (Jesson & Lacey, 2006 p. 142 – 143) The narrative critical literature reviews summarise the available literature and provide a feasible origin of back-ground information. Haneline supports that in the healthcare industry, those reviews are more useful since they are simpler and are usually written by an expert. (2007, p. 20)
4.2. Systematic Review
A systematic review is a research method, which acquires its results from the critical interpretation of the existed literature. (Jesson & Lacey, 2006 p. 145) Systematic review’s target is to apply an explicit method, so as it can be replicated, in parliamentary procedure to place all the relevant research literature to answer a given question without prejudice at any phase of the inspection procedure. (McGowan, 2012 p. 589) As Woods claimed a systematic review is “researchers aim for a way to control madness” (1999, p. 10)
Haneline provides a comprehensive definition of a systematic review: “A systematic review is defined as the application of scientific strategies that limit bias to the systematic assembly, critical appraisal and synthesis of all relevant studies of a specific topic”. (2007, p. 19)
The major three differences between narrative literature reviews and systematic reviews are identified by Haneline (2007 p.19), Kowalczyk & Truluck (2010 p. 220 – 222) and Jesson & Lacey (2006 p. 145).
Initially, the search process includes thorough and definitive search strategies. The author openly states the inclusion criteria, as described in Section 3.1, and performs an attempt to cover all published as well as material from the Internet. Those actions prevent bias behalf of the author. Furthermore, there should be strict and clear criteria for data extraction from studies, and for estimating the quality of the evidence on which they are based. (Jesson & Lacey, 2006 p. 146). Haneline suggests that the data evaluation should be performed by two or more researchers (2007 p.21), and Kowalczyk & Truluck propose that a third reviewer should be consulted in case of disagreement between the researchers. (2010 p. 220) Jesson & Lacey identified as the final difference between a systematic and a narrative review, the “depth of understanding the reviewer needs of each report”. The reviewer has to be capable to critically assess each study in a systematic review (2006 p. 148)
Although systematic review is praised by the majority of the literature, there have been identified a few cases that a rigorous approach of a rapid literature review considered more successful, as it was stated in Section 1. Apart from the case of the rapid literature review, there has been a few papers identified that are illustrating some flaws of a systematic literature review.
Wong, Greenhalgh, Westhorp, Buckingham & Pawson, identify an alternative systematic review method, namely “realist synthesis”, which they argue it offers better understanding between context and outcome by using the realist philosophy. (2013, p. 1005-1006) Mallidou also mentions the realist synthesis as a systematic review variation. Likewise she defines the scoping reviews as a child product of a systematic review with aim to “quickly map the key concepts supporting a research area and can be attempted as stand-alone small projects” (2014, p. 36) Moreover, another problem is determined by May, Lu & Xue, which is related to the volume and the mean of the available bibliography. As they state, the majority of the papers that they need to review is either in printed format or the collection size is enormous; as a result, they suggest a new algorithm on how to quickly evaluate available resources for systematic searching. (2012 p. 1102). Finally, the problem of the volume of the related information is also highlighted by Gough, Oliver & Thomas; followed by a “methodological issue”, which is the lack of definite terminology to describe the methods, issue that the vast majority of reviewed papers also mention. (2012 p. 10-12)
Meta-analysis literature review is the newest and arguably the most debatable methodology of performing a literature review, on the grounds that the reviewed literature appears to be divided in half.
There have been identified seven papers that refer to meta-analysis methodology, four of those are describing the methodology as stand-alone and three of them are supporting that meta-analysis is a follow-up procedure of a systematic literature review. This diversity does not seem to be affected by the scientific principle of those papers. It is quite clear and simply stated that a meta-analysis review is a complementary way that the researcher uses to highlight and contrast the findings of the main topic area (Wong, Greenhalgh, Westhorp, Buckingham & Pawson, 2013 p. 987) and (Booth, 2006 p. 422). Although the time that systematic reviews were developing, meta-analysis was a discrete method of bibliographic synthesis, the field of systematic literature reviews developed and recognised meta-analysis as a valid literature review design methodology. (Marshall, Goldbart, Pickstone, & Roulstone, 2011 p. 262)
Nowadays, meta-analysis is often used in health and clinical research, evaluating statistically quantitative research studies. (Jesson & Lacey, 2006 p. 146) It is still commonly follows a systematic review, and provides a more objective review of the literature.
A useful definition of a meta-analysis is provided by Mallidou: “Meta-analysis is a transparent, objective and replicable statistical method that aims to combine and compare evidence from independent, relevant and quantitative studies and supports evidence-based policy” (2014, p. 37)
An argument in favour of narrative reviews instead of meta-analysis as well as the systematic literature review is the public familiarity with and the simplicity of the narrative reviews. (Borenstein, 2009 p. 385) On the other hand, narrative reviews suffer from the same problems that were mentioned in their comparison with the systematic literature reviews, in the previous Section 4.2. Borenstein argues that many of the flaws of meta-analyses, are based on misuse of the meta-analysis technique. (2009, p.385) Another argument in favour of meta-analysis is that it provides a clear, objective an easy to replicate method, in contrast with systematic literature reviews in which there is subjectivity in setting the research and inclusion rules. (Mallidou 2014, p.37)
This paper, provides a detailed insight on literature reviews. In summary, all methods of designing and writing a literature review are equally important, depending on the research field, time limitations as well as the target audience, those methods should vary. As it is mentioned in the previous sections, a narrative literature review might be better for a simple theme or maybe a preparatory literature review and a meta-analysis to shine on statistical data analysis, but it is equally important to make sure that all the principles of each literature review methodology are applied correctly.
Writing a literature review can be challenging especially to inexperience authors. However, the ability to critically appraise literature is a skill that all researchers should have. It is desired that this literature review, will help evaluating and clarifying the criteria of a successful literature review.
(n.d.). Retrieved 22 November 2015, from http://www.scs.ryerson.ca/aferworn/courses/CP8101/CLASSES/ConductingLiteratureReview.pdf
Borenstein, M. (2009). Introduction to meta-analysis. [Electronic resource]. Chichester, West Sussex, U.K. ; Hoboken : John Wiley & Sons, 2009.
Bond, A. (1999). Your Masters Thesis: Helping You to Achieve (In-Focus, Studymates). (G. Lawler, Ed.) (2nd ed.). United Kingdom: Studymates.
Booth, A. (2006). ‘Brimful of STARLITE’: toward standards for reporting literature searches. Journal Of The Medical Library Association, 94(4), 421-e205 10p.
Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: evaluation of three strategies. Journal Of Advanced Nursing, 57(1), 95-100 6p. doi:10.1111/j.1365-2648.2006.04083.x
Furseth, I. E., & Everett, E. L. (2013). Doing your masters Dissertation: From start to finish. London, England: SAGE Publications.
Gough, D., Oliver, S., & Thomas, J. (Eds.). (2012). An introduction to systematic reviews. United Kingdom: SAGE Publications.
Haneline, M. (2007). Understanding literature review designs. Journal Of The American Chiropractic Association, 44(3), 19-23 5p.
Harker, J., & Kleijnen, J. (2012). What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. International Journal Of Evidence-Based Healthcare (Wiley-Blackwell), 10(4), 397-410 14p. doi:10.1111/j.1744-1609.2012.00290.x
Hart, C. (1999). Doing a Literature Review: Releasing the Social Science Research Imagination (Published in association with The Open University) (1st ed.). United Kingdom: Sage Publications.
Jesson, J., & Lacey, F. (2006). How to do (or not to do) a critical literature review. Pharmacy Education, 6(2), 139-148. doi:10.1080/15602210600616218
Kumar, R. (2014). Research methodology: A step-by-step guide for beginners (4th ed.). London: SAGE Publications : SAGE Publications.
Kowalczyk, N., & Truluck, C. (2013). Literature Reviews and Systematic Reviews: What Is the Difference?. Radiologic Technology, 85(2), 219-222 4p.
Levitt, C., Shaw, E., Wong, S., Kaczorowski, J., Springate, R., Sellors, J., & Enkin, M. (2004). Systematic review of the literature on postpartum care: methodology and literature search results. Birth: Issues In Perinatal Care, 31(3), 196-202 7p.
MacNealy, M. S. (1998). Strategies for empirical research in writing(1st ed.). Boston: Allyn and Bacon.
Mallidou, A. (2014). Mapping the landscape of knowledge synthesis. Nursing Management – UK, 21(5), 30-39 10p. doi:10.7748/nm.21.5.30.e1242
Marshall, G. (2010). Writing… a literature review…third in a series. Synergy: Imaging & Therapy Practice, 20-23 4p.
Marshall, J., Goldbart, J., Pickstone, C., & Roulstone, S. (2011). Application of systematic reviews in speech-and-language therapy. International Journal Of Language & Communication Disorders, 46(3), 261-272 12p. doi:10.3109/13682822.2010.497530
May, B. H., Lu, C., & Xue, C. C. (2012). Collections of Traditional Chinese Medical Literature as Resources for Systematic Searches. Journal Of Alternative & Complementary Medicine, 18(12), 1101-1107 7p. doi:10.1089/acm.2011.0587
McAlister, F. A., & Clark, H. D. (1999). The medical review article revisited: Has the science improved?. Annals Of Internal Medicine, 131(12), 947-951.
McCrae N., Blackstock M. & Purssell E. (2015) Eligibility criteria in systematic reviews: A methodological review. International Journal of Nursing Studies, 52(7), 1269-1276 8p. doi:10.1016/j.ijnurstu.2015.02.002
McGowan, L. (2012). Systematic reviews: the good, the not so good and the good again. British Journal Of Midwifery, 20(8), 588-592 5p.
Mongan-Rallis, H. (2014, November 21). Guidelines for writing a literature review. Retrieved December 29, 2015, from http://www.duluth.umn.edu/~hrallis/guides/researching/litreview.html
Ridley, D. (2012). The literature review: A step-by-step guide for students (2nd ed.). London: SAGE Publications.
Smallbone, T., & Quinton, S. (2011). A three-stage framework for teaching literature reviews: A new approach. International Journal Of Management Education (Oxford Brookes University), 9(4), 1-11. doi:10.3794/ijme.94.337
Thody, A. (2006). Writing and Presenting Research (Sage Study Skills Series) (1st ed.). London: Sage Publications.
Vogt, P. W., Gardner, D. C., & Haeffele, L. M. (2012). When to use what research design. New York, NY: Guilford Publications.
Watt A., Cameron A., Sturm L., Lathlean T., Babidge W., Blamey S., Facey K., Hailey D., Norderhaug I. & Maddern G. (2008). Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment. International Journal of Technology Assessment in Health Care, 24(2), 133-139. doi:10.1017/S0266462308080185.
Walsh, D., & Downe, S. (2005). Meta-synthesis method for qualitative research: a literature review. Journal Of Advanced Nursing, 50(2), 204. doi:10.1111/j.1365-2648.2005.03380.x
Wisker, G. (2008). Postgraduate research handbook, 2nd edition: Succeed with your MA, MPhil, EdD and PhD (Palgrave study guides) (2nd ed.). Basingstoke, Hampshire: Palgrave Macmillan.
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: meta-narrative reviews. Journal Of Advanced Nursing, 69(5), 987-1004 18p. doi:10.1111/jan.12092
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. Journal Of Advanced Nursing, 69(5), 1005-1022 18p. doi:10.1111/jan.12095
Woods, P. (1999). Successful Writing For Qualitative Researchers. Routledge.
Wright, S., & McSherry, W. (2013). A systematic literature review of Releasing Time to Care: The Productive Ward. Journal Of Clinical Nursing, 22(9/10), 1361-1371 11p. doi:10.1111/jocn.12074
|Paper||Relevance||Lit Review Methodology||Lit Review Definition||Lit Review as Main Subject||notes|
|What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments||yes||rapid review||yes||yes|
|Brimful of STARLITE: toward standards for reporting literature searches||yes||systematic review||no||no|
|Analysis of the reporting of search strategies in Cochrane systematic reviews||yes||systematic review||yes||yes|
|Literature Reviews and Systematic Reviews: What Is the Difference?||yes||systematic review and meta-analysis||yes||yes|
|RAMESES publication standards: realist syntheses||yes||realist synthesis as an alternative systematic review||no||no|
|Understanding Literature Review Designs||yes||all 3 types||yes||yes|
|RAMESES publication standards: meta-narrative reviews||yes||meta-narrative review||no||yes|
|Application of systematic reviews in speech-and-language therapy||yes||systematic review||no||no|
|Meta-synthesis method for qualitative research: a literature review||yes||meta-analysis||yes||no|
|A Review of Preanalytical Factors Affecting Molecular, Protein, and Morphological Analysis of Formalin-fixed, Paraffin-Embedded (FFPE) Tissue||no||meta-analysis||no||no|
|Mapping the landscape of knowledge synthesis||yes||no||no||no|
|A three-stage framework for teaching literature reviews: A new approach||yes||systematic review||yes||yes|
|Electronic searching to locate qualitative research: evaluation of three strategies||yes||systematic review||no||no|
|Swab and instrument count practice: ways to enhance patient safety||no||no|
|The Development of Evidence-Based Clinical Simulation Scenarios: Guidelines for Nurse Educators||no||no|
|What about Plan Evaluation? Integrating Evaluation in Urban Planning Studio’s Pedagogy||no||no|
|An ecological approach to health promotion in remote Australian Aboriginal communities||no||no|
|The scoping study in physical therapy: application of traditional systematic review guidelines to an emerging methodology||no||no|
|Rethinking health research capacity strengthening||no||no|
|State-of-the-Evidence Reviews: Advantages and Challenges of Including Grey Literature||no||no|
|Designing an Acupuncture Study: II. The Nationwide, Randomized, Controlled German Acupuncture Trials on Low-Back Pain and Gonarthrosis||no||no|
|Collections of Traditional Chinese Medical Literature as Resources for Systematic Searches||no||no|
|The Development of Canadian Clinical Practice Guidelines: a literature review and synthesis of findings||no||not defined||no||no|
|Systematic Review of the Literature on Postpartum Care: Methodology and Literature Search Results||no||systematic review||no||no|
|Searching for evidence||no||no|
|Arterial transducer placement and cerebral perfusion pressure monitoring: a discussion||no||systematic review||no|
|Tailoring Interventions: Examining the Evidence and Identifying Gaps||no||systematic review||no|