The aim of this study is to recognise key criteria relating to the development of an algorithm that would allow a researcher focused on simulation modelling under uncertainty to rapidly identify literature relevant to this research, in a systematic, methodological, replicable and accurate manner. This research study has been undertaken as part of the research team of Peter B., Penny R., Peter I. and I. The results of this study will help determine how to produce a systematic literature review in a rapid manner on the task of research for academic literature.
The existing research methodologies are based on inclusion, exclusion, or both criteria that are used to identify relevant papers from huge databases. One of the key elements of a systematic literature review methodology, is the ability to search “every” database and “every” literature resource, online or in paper. However, a problem has been identified by May, Lu & Xue, which is related to the volume and the mean of the available bibliography. As they state, the majority of the papers that they need to review is either in printed format or the collection size is enormous; as a result, they suggest a new algorithm on how to quickly evaluate available resources for systematic searching. (2012 p. 1102). Finally, the problem of the volume of the related information is also highlighted by Gough, Oliver & Thomas; followed by a “methodological issue”, which is the lack of definite terminology to describe the methods, issue that the vast majority of reviewing papers also mention. (2012 p. 10-12)
It is hoped that the end result of this study will be a concise, systematic, replicable and accurate algorithm that will assist the researchers of the team in rigorously identifying the relevant literature for simulation modelling under uncertainty.
2. Research Aims
This is a continuation of an existing project, as it is mentioned in Section 1. My motivation and the fundamental goal of this project, is to develop an algorithm for the following scenario. There are hundreds of relevant research papers for performing a literature review on simulation modelling under uncertainty. The aim of this algorithm should be to determine criteria, categories and groups of research papers that should be included, excluded, and being explored further.
- To identify key elements, barriers and words for research algorithms/methodologies in simulation modelling under uncertainty.
- To develop an algorithm that identifies papers that satisfy one of the three categories: inclusion, exclusion and further exploration criteria.
- The above algorithm should suffice the criteria of being used in a Systematic Literature review as it is defined by (Marshall, 2010 p. 22)
3. Research Proposal
The question this study intents to answer is: What is the most efficient literature research algorithm for simulation modelling under uncertainty? This question has been given limited consideration in the academic literature and is an important query for numerous reasons. During the recent years there has been an explosion of available literature, of high quality, via the online databases, which can be accessed everywhere in the world. The need to be able to perform a rigorous but also efficient, accurate and dynamic research in order to find relevant research material is important. Currently, there is a growing literature that notices the need for a research algorithm that would assist researchers into identifying papers that are relevant to their research. There are no clear guidelines (Harker & Kleijnen, 2012, p. 398)
Reviewing the literature is an ongoing process, throughout the research lifecycle. Ridley correlates that continuing process with a “recursive literature review” (2012 p. 19). An accord exists in literature that the first step into writing a good literature review is to define the research question, to which the review produced should be relevant. (Kumar, 2014 p. 60)
3.1. Research Strategy and Design
A research design performs two basic roles in research: (1) describe the procedures for undertaken the study; and (2) to verify that “in case of causality, the independent variable has the highest chance affect the dependent variable, unrelated to the extraneous and chance variables”. (Kumar, 2014 p. 129)
According to Vogt, Gardner & Haeffele, (2012 p. 106) and Kumar (2014, p. 132-133) qualitative research is most commonly used in research related to a group of people or population, with flexible and non-sequential designs. However, in quantitative research the structure of the research is more rigid and predetermined, in order to produce valid, reliable and replicable results.
The proposed research methodology is a qualitative study, using secondary sources search for data collection and data analysis using a Microsoft Excel and SPSS.
The main focus of this study is to propose an algorithm that will suggest relevant literature for the predefined field of simulation modelling under uncertainty. All relevant literature as specified by Spagnoletti, Za & Winter with the addition of the recent years 2013-2016 under the same criteria, constitutes the main study population, as it is impossible to researcher to identify every relevant academic paper from every source, due to the time constraints (2013). The academic papers will be separated and categorized based on the relevance, quality of references, and quantity of citations, inclusion and exclusion criteria as well as further criteria defined at a later stage of the research. The first step of the research would be to read through all the titles and abstracts of the papers to judge relevance. Followed by the second stage of further studying and categorizing the relevant papers. At this stage a systematic literature review should be produced, with elements of meta-analysis due to the quantitative nature of the produced results. The results of the literature review will be compared with the bibliographic results produced by Spagnoletti, Za & Winter (2013), for completeness.
The expected result of this action is to perform a statistical analysis, as it is described in Section 3.5 and identify common elements of the papers that will allow us to further our search and include more databases and resources as well as develop the final algorithm.
3.2. Data Collection
The data collection for this study will be performed in the manner of a secondary source search (Kumar, 2014 p. 196). This will have a huge impact on the quality of the data gathered, due to the fact that there are many publications that do not define with clarity their research methodology or their inclusion criteria; thus the validity of their findings could be questioned. Haneline suggests that the data evaluation should be performed by two or more researchers (2007 p. 21), due to the fact that this study has been undertaken as part of a much larger team, the proposed evaluation is possible. The accepted data will be part of the final proposed algorithm, with a summary of the discarded data as a statistical quantity. A form will be developed for gathering this data in a spreadsheet format.
3.2.1. Data Type
The data that are mentioned above will have the form of numerical and categorical data. For example:
- Does this paper have clearly identified inclusion criteria? The possible answers could be yes and no.
- Is this paper relevant to simulation modelling? The possible answers could again be yes and no.
- How many references does this paper have or how many times has this paper cited? The possible answers are integer numbers.
3.3. Ethical And Confidentiality Issues
No ethical and confidentiality issues have been identified for this study.
3.4. Theoretical Perspectives
The review of the literature depends on four key factors, namely methods to search the literature, quality of literature being searched, methodology of organizing the research results and referencing. It is typical amongst researchers to provide a few advices and common criteria of what they regard as a good literature review. (Mallidou, 2014 p. 32 and Kumar, 2014 p. 60) However, according to Kowalczyk & Truluck, a good literature review is not just a collection of literature numbered and summarized, but a critical evaluation of the collected literature (2013 p. 219). An impressive fact, is that both Marshall (2010) as well as Wong, Greenhalgh, Westhorp, Buckingham & Pawson, (2013) put a lot of effort to explain how to critically evaluate the resourced material, but they do not mention acceptable sources and methodologies for collecting those. Actually, there are only a few authors that mention the difference between the types of literature and what can be considered a “good” literature source; the vast majority of whom, they also support a variety of literature review methodologies and do not focus only on systematic literature reviews and their variations.
The main sources of identifying literature are books, journals, conference papers and the Internet. (Kumar, 2014 p.60) Importantly, the main criteria for appraising the collected literature focusing on the above sources are described by Mallidou (2014, p.33) and they can be encapsulated as the use of researcher’s experience and judgement, which can lead to bias, is unavoidable; therefore the application of a generic standardised instrument or algorithm would minimise the bias (Ridley, 2012) and (Gough, Oliver & Thomas, 2012).
The element of bias as well as the personal experience and judgements lead to the final and arguably the most important fact of a literature review search, which is the inclusion criteria. Literature reviews that have numbered and clearly stated inclusion criteria are considered to be more accurate by the majority of the readers. (Marshall, 2010 p. 20) Although this can be considered true for some literature reviews, McCrae, Blackstock & Purssell support that plethora of reviews specify exclusion criteria instead of inclusion, (2015, p. 1272) argument that is also supported by Wright & McSherry (2013, p. 1362). The incorporation of the exclusion criteria instead of the inclusion, is considered “more useful by cause of the eligibility being framed negatively, instead of a positive specification with clearly stated conditions”. (McCrae, Blackstock & Purssell, 2015 p. 1272) An interesting observation in regard to which type of criteria should a writer using is able to be made here, since the majority of authors do not differentiate between “exclusion or inclusion” criteria, they however, focus towards clarity and simple definition of one of those criteria categories. (Flemming & Briggs, 2007 p. 97)
Because of the time constraints, it is required a quite narrow scope of this study to be defined. The proposed framework should demonstrate a deep understanding of theories and concepts related to literature review, inclusion, exclusion criteria as well as simulation models that can be discussed focusing on ambiguous and uncertain problem spaces and information system as a discipline.
3.5. Methods for Analysis
Analysis of the data consists of many steps. The initial step is gathering the data as described in Section 3.2; following that the data need to be processed and “cleaned” or edited, to make sure that there are no wrongly classifications, errors and misunderstandings on the data derived by the literature. The editing of the data one by one seem possible, due to a limited volume of data that will be generated.
The next step after editing the data is described in literature as coding. The type of coding depends on the type of gathering data. The categorical data that will be produced in this study will be coded by assigning a numerical value in each category. (Kumar, 2014 p. 269)
The data analysis and the statistical procedures will be completed by computer, using the SPSS and Microsoft Excel software.
Even though the total number of exact procedures will be agreed upon by the team at a later stage of this study, a few can be proposed at this stage. For example, regression analysis (between the titles of related papers and abstracts) as well as multiple regression analysis (between the existence of clearly defined methodology, inclusion criteria and relevance in simulation modeling) and correlation (between abstracts of related papers and key concepts of the research) will be used.
This section presents my schedule for performing the proposed research. This research culminates in a formal report that will be completed by September 14, 2016; in order for me to reach this goal, the schedule presented in Figure 1 will be followed.
Figure 1 Research Gantt chart
The tasks that are highlighted with red colour in Figure 1, are the initial milestones of the research.
Provided that I am able to obtain all my sources for the literature review and the consecutive research from the library of the University of Portsmouth via Discovery search engine, with the help of the librarians, there is no appreciable cost related to performing this research. I am also a member of IET and ResearchGate, therefore their databases can be accessed to search for more articles, without any extra cost.
Furthermore, I am located in Portsmouth, as all the members of the research team, which this research paper is part of, and no additional costs for meeting and consulting with them can be determined. As for the technological resources, a laptop with Microsoft Word, Visio and Excel as well as a USB stick are the only reasonable resources that can be identified, due to the fact that the research project does not involve an actual artefact; those technological resources are available without any cost in the University of Portsmouth library.
The only costs that can be identified, will be minor, such as copying and printing key articles as well as printing and binding the final research. A logical estimate can be, that these tasks can be performed for under £20.
Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: evaluation of three strategies. Journal Of Advanced Nursing, 57(1), 95-100 6p. doi:10.1111/j.1365-2648.2006.04083.x
Gough, D., Oliver, S., & Thomas, J. (Eds.). (2012). An introduction to systematic reviews. United Kingdom: SAGE Publications.
Haneline, M. (2007). Understanding literature review designs. Journal Of The American Chiropractic Association, 44(3), 19-23 5p.
Harker, J., & Kleijnen, J. (2012). What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. International Journal Of Evidence-Based Healthcare (Wiley-Blackwell), 10(4), 397-410 14p. doi:10.1111/j.1744-1609.2012.00290.x
Kumar, R. (2014). Research methodology: A step-by-step guide for beginners (4th ed.). London: SAGE Publications : SAGE Publications.
Kowalczyk, N., & Truluck, C. (2013). Literature Reviews and Systematic Reviews: What Is the Difference?. Radiologic Technology, 85(2), 219-222 4p.
Mallidou, A. (2014). Mapping the landscape of knowledge synthesis. Nursing Management – UK, 21(5), 30-39 10p. doi:10.7748/nm.21.5.30.e1242
Marshall, G. (2010). Writing… a literature review…third in a series. Synergy: Imaging & Therapy Practice, 20-23 4p.
May, B. H., Lu, C., & Xue, C. C. (2012). Collections of Traditional Chinese Medical Literature as Resources for Systematic Searches. Journal Of Alternative & Complementary Medicine, 18(12), 1101-1107 7p. doi:10.1089/acm.2011.0587
McCrae N., Blackstock M. & Purssell E. (2015) Eligibility criteria in systematic reviews: A methodological review. International Journal of Nursing Studies, 52(7), 1269-1276 8p. doi:10.1016/j.ijnurstu.2015.02.002
Ridley, D. (2012). The literature review: A step-by-step guide for students (2nd ed.). London: SAGE Publications.
Spagnoletti, P., Za, S., & Winter, R. (2013). Exploring Foundations for Using Simulations in IS Research. In Proceedings of the International Conference On Information Systems, ICIS 2013. AIS Electronic Library (AISeL): Association for Information Retrieved from: Systems.https://www.researchgate.net/profile/Paolo_Spagnoletti/publication/258299724_Exploring_Foundations_for_Using_Simulations_in_IS_Research/links/0deec527bd642aa047000000.pdf
Vogt, P. W., Gardner, D. C., & Haeffele, L. M. (2012). When to use what research design. New York, NY: Guilford Publications.
Wright, S., & McSherry, W. (2013). A systematic literature review of Releasing Time to Care: The Productive Ward. Journal Of Clinical Nursing, 22(9/10), 1361-1371 11p. doi:10.1111/jocn.12074
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. Journal Of Advanced Nursing, 69(5), 1005-1022 18p. doi:10.1111/jan.12095