Extensive Review of Literature Qualitative or Quantitative Research
- Methodology
- Open Access
- Published:
Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research
Systematic Reviews volume 5, Article number:172 (2016) Cite this article
Abstract
Background
Overviews of methods are potentially useful ways to increment clarity and raise collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that need unique review procedures. The purpose of this paper is to initiate give-and-take nearly what a rigorous systematic approach to reviews of methods, referred to here as systematic methods overviews, might wait like by providing tentative suggestions for approaching specific challenges likely to be encountered. The guidance offered here was derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research.
Results
The guidance is organized into several principles that highlight specific objectives for this type of review given the common challenges that must be overcome to achieve them. Optional strategies for achieving each principle are besides proposed, along with word of how they were successfully implemented in the overview on sampling. We draw 7 paired principles and strategies that accost the post-obit aspects: delimiting the initial set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology used to describe specific methods topics, and generating rigorous verifiable analytic interpretations. Since a broad aim in systematic methods overviews is to describe and translate the relevant literature in qualitative terms, we advise that iterative decision making at diverse stages of the review process, and a rigorous qualitative approach to analysis are necessary features of this review type.
Conclusions
We believe that the principles and strategies provided here will exist useful to anyone choosing to undertake a systematic methods overview. This paper represents an initial effort to promote high quality critical evaluations of the literature regarding problematic methods topics, which have the potential to promote clearer, shared understandings, and accelerate advances in research methods. Farther work is warranted to develop more definitive guidance.
Peer Review reports
Background
While reviews of methods are not new, they represent a distinct review type whose methodology remains relatively under-addressed in the literature despite the clear implications for unique review procedures. One of few examples to draw it is a affiliate containing reflections of two contributing authors in a volume of 21 reviews on methodological topics compiled for the British National Health Service, Wellness Technology Assessment Program [ane]. Notable is their observation of how the differences between the methods reviews and conventional quantitative systematic reviews, specifically attributable to their varying content and purpose, take implications for defining what qualifies as systematic. While the authors draw general aspects of "systematicity" (including rigorous application of a methodical search, abstraction, and analysis), they also describe a high degree of variation within the category of methods reviews itself and and so offer piddling in the way of concrete guidance. In this paper, we nowadays tentative concrete guidance, in the course of a preliminary set of proposed principles and optional strategies, for a rigorous systematic approach to reviewing and evaluating the literature on quantitative or qualitative methods topics. For purposes of this article, we have used the term systematic methods overview to emphasize the notion of a systematic approach to such reviews.
The conventional focus of rigorous literature reviews (i.e., review types for which systematic methods take been codified, including the various approaches to quantitative systematic reviews [2–4], and the numerous forms of qualitative and mixed methods literature synthesis [5–10]) is to synthesize empirical research findings from multiple studies. By dissimilarity, the focus of overviews of methods, including the systematic approach we advocate, is to synthesize guidance on methods topics. The literature consulted for such reviews may include the methods literature, methods-relevant sections of empirical enquiry reports, or both. Thus, this newspaper adds to previous piece of work published in this journal—namely, recent preliminary guidance for conducting reviews of theory [11]—that has extended the application of systematic review methods to novel review types that are concerned with discipline matter other than empirical research findings.
Published examples of methods overviews illustrate the varying objectives they can have. One objective is to establish methodological standards for appraisal purposes. For instance, reviews of existing quality appraisal standards take been used to propose universal standards for appraising the quality of primary qualitative enquiry [12] or evaluating qualitative research reports [13]. A second objective is to survey the methods-relevant sections of empirical research reports to establish current practices on methods use and reporting practices, which Moher and colleagues [fourteen] recommend equally a means for establishing the needs to be addressed in reporting guidelines (see, for instance [fifteen, 16]). A tertiary objective for a methods review is to offering clarity and enhance collective agreement regarding a specific methods topic that may exist characterized past ambiguity, inconsistency, or a lack of comprehensiveness inside the available methods literature. An instance of this is a overview whose objective was to review the inconsistent definitions of intention-to-treat analysis (the methodologically preferred arroyo to analyze randomized controlled trial data) that have been offered in the methods literature and propose a solution for improving conceptual clarity [17]. Such reviews are warranted because students and researchers who must larn or utilize inquiry methods typically lack the time to systematically search, call up, review, and compare the available literature to develop a thorough and critical sense of the varied approaches regarding certain controversial or ambiguous methods topics.
While systematic methods overviews, equally a review type, include both reviews of the methods literature and reviews of methods-relevant sections from empirical study reports, the guidance provided here is primarily applicable to reviews of the methods literature since information technology was derived from the experience of conducting such a review [18], described below. To our knowledge, there are no well-developed proposals on how to rigorously conduct such reviews. Such guidance would have the potential to improve the thoroughness and credibility of disquisitional evaluations of the methods literature, which could increase their utility as a tool for generating understandings that accelerate inquiry methods, both qualitative and quantitative. Our aim in this paper is thus to initiate discussion well-nigh what might establish a rigorous approach to systematic methods overviews. While nosotros hope to promote rigor in the conduct of systematic methods overviews wherever possible, we do not wish to suggest that all methods overviews need be conducted to the same standard. Rather, nosotros believe that the level of rigor may demand to be tailored pragmatically to the specific review objectives, which may not always justify the resource requirements of an intensive review process.
The instance systematic methods overview on sampling in qualitative research
The principles and strategies we propose in this paper are derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research [18]. The primary objective of that methods overview was to bring clarity and deeper agreement of the prominent concepts related to sampling in qualitative enquiry (purposeful sampling strategies, saturation, etc.). Specifically, we interpreted the available guidance, commenting on areas lacking clarity, consistency, or comprehensiveness (without proposing whatsoever recommendations on how to do sampling). This was achieved past a comparative and critical analysis of publications representing the most influential (i.e., highly cited) guidance across several methodological traditions in qualitative enquiry.
The specific methods and procedures for the overview on sampling [18] from which our proposals are derived were adult both afterward soliciting initial input from local experts in qualitative inquiry and an adept wellness librarian (KAM) and through ongoing conscientious deliberation throughout the review procedure. To summarize, in that review, nosotros employed a transparent and rigorous approach to search the methods literature, selected publications for inclusion according to a purposeful and iterative process, bathetic textual data using structured abstraction forms, and analyzed (synthesized) the data using a systematic multi-stride arroyo featuring brainchild of text, summary of information in matrices, and analytic comparisons.
For this article, we reflected on both the problems and challenges encountered at different stages of the review and our means for selecting justifiable procedures to deal with them. Several principles were then derived by because the generic nature of these problems, while the generalizable aspects of the procedures used to address them formed the basis of optional strategies. Farther details of the specific methods and procedures used in the overview on qualitative sampling are provided below to illustrate both the types of objectives and challenges that reviewers will likely demand to consider and our arroyo to implementing each of the principles and strategies.
Organisation of the guidance into principles and strategies
For the purposes of this article, principles are general statements outlining what we advise are of import aims or considerations within a particular review process, given the unique objectives or challenges to exist overcome with this blazon of review. These statements follow the full general format, "considering the objective or challenge of X, we propose Y to be an of import aim or consideration." Strategies are optional and flexible approaches for implementing the previous principle outlined. Thus, generic challenges give ascension to principles, which in turn requite rise to strategies.
We organize the principles and strategies below into 3 sections corresponding to processes feature of almost systematic literature synthesis approaches: literature identification and choice; data abstraction from the publications selected for inclusion; and assay, including critical appraisal and synthesis of the abstracted data. Inside each section, we as well describe the specific methodological decisions and procedures used in the overview on sampling in qualitative inquiry [18] to illustrate how the principles and strategies for each review procedure were applied and implemented in a specific case. We expect this guidance and accompanying illustrations volition be useful for anyone considering engaging in a methods overview, particularly those who may be familiar with conventional systematic review methods but may not yet appreciate some of the challenges specific to reviewing the methods literature.
Results and word
Literature identification and choice
The identification and option procedure includes search and retrieval of publications and the development and application of inclusion and exclusion criteria to select the publications that volition exist abstracted and analyzed in the final review. Literature identification and selection for overviews of the methods literature is challenging and potentially more resources-intensive than for most reviews of empirical research. This is true for several reasons that we describe below, alongside discussion of the potential solutions. Additionally, nosotros suggest in this section how the selection procedures can be called to match the specific analytic arroyo used in methods overviews.
Delimiting a manageable prepare of publications
Ane attribute of methods overviews that can make identification and selection challenging is the fact that the universe of literature containing potentially relevant information regarding most methods-related topics is expansive and often unmanageably then. Reviewers are faced with two big categories of literature: the methods literature, where the possible publication types include periodical articles, books, and book capacity; and the methods-relevant sections of empirical study reports, where the possible publication types include journal articles, monographs, books, theses, and conference proceedings. In our systematic overview of sampling in qualitative research, exhaustively searching (including retrieval and outset-pass screening) all publication types beyond both categories of literature for information on a unmarried methods-related topic was too burdensome to exist feasible. The following proposed principle follows from the need to delimit a manageable set of literature for the review.
Principle #i:
Considering the broad universe of potentially relevant literature, nosotros propose that an important objective early in the identification and selection stage is to circumscribe a manageable set of methods-relevant publications in accordance with the objectives of the methods overview.
Strategy #i:
To limit the prepare of methods-relevant publications that must be managed in the selection process, reviewers have the option to initially review simply the methods literature, and exclude the methods-relevant sections of empirical study reports, provided this aligns with the review'due south particular objectives.
Nosotros propose that reviewers are justified in choosing to select only the methods literature when the objective is to map out the range of recognized concepts relevant to a methods topic, to summarize the virtually authoritative or influential definitions or meanings for methods-related concepts, or to demonstrate a problematic lack of clarity regarding a widely established methods-related concept and potentially brand recommendations for a preferred approach to the methods topic in question. For example, in the case of the methods overview on sampling [18], the primary aim was to ascertain areas lacking in clarity for multiple widely established sampling-related topics. In the review on intention-to-treat in the context of missing outcome data [17], the authors identified a lack of clarity based on multiple inconsistent definitions in the literature and went on to recommend separating the event of how to handle missing outcome data from the upshot of whether an intention-to-treat analysis can be claimed.
In contrast to strategy #1, it may exist appropriate to select the methods-relevant sections of empirical written report reports when the objective is to illustrate how a methods concept is operationalized in inquiry practice or reported past authors. For example, one could review all the publications in ii years' worth of issues of five high-impact field-related journals to answer questions nearly how researchers describe implementing a particular method or approach, or to quantify how consistently they define or report using it. Such reviews are often used to highlight gaps in the reporting practices regarding specific methods, which may be used to justify items to accost in reporting guidelines (for example, [14–16]).
It is worth recognizing that other authors have advocated broader positions regarding the scope of literature to exist considered in a review, expanding on our perspective. Suri [10] (who, like us, emphasizes how dissimilar sampling strategies are suitable for different literature synthesis objectives) has, for example, described a two-stage literature sampling process (pp. 96–97). First, reviewers utilize an initial approach to conduct a wide overview of the field—for reviews of methods topics, this would entail an initial review of the research methods literature. This is followed by a 2d more than focused stage in which applied examples are purposefully selected—for methods reviews, this would involve sampling the empirical literature to illustrate key themes and variations. While this approach is seductive in its capacity to generate more in depth and interpretive analytic findings, some reviewers may consider it as well resource-intensive to include the 2nd step no matter how selective the purposeful sampling. In the overview on sampling where we stopped after the first stage [18], nosotros discussed our selective focus on the methods literature every bit a limitation that left opportunities for farther analysis of the literature. We explicitly recommended, for case, that theoretical sampling was a topic for which a future review of the methods sections of empirical reports was justified to reply specific questions identified in the chief review.
Ultimately, reviewers must make pragmatic decisions that residuum resource considerations, combined with informed predictions almost the depth and complexity of literature available on their topic, with the stated objectives of their review. The remaining principles and strategies utilise primarily to overviews that include the methods literature, although some aspects may exist relevant to reviews that include empirical study reports.
Searching beyond standard bibliographic databases
An of import reality affecting identification and option in overviews of the methods literature is the increased likelihood for relevant publications to be located in sources other than periodical articles (which is commonly not the case for overviews of empirical research, where periodical articles generally represent the primary publication type). In the overview on sampling [18], out of 41 full-text publications retrieved and reviewed, only 4 were periodical articles, while 37 were books or book chapters. Since many books and book chapters did not exist electronically, their full text had to be physically retrieved in hardcopy, while 11 publications were retrievable but through interlibrary loan or buy request. The tasks associated with such retrieval are substantially more than time-consuming than electronic retrieval. Since a substantial proportion of methods-related guidance may exist located in publication types that are less comprehensively indexed in standard bibliographic databases, identification and retrieval thus become complicated processes.
Principle #two:
Considering that of import sources of methods guidance can be located in not-periodical publication types (e.g., books, book capacity) that tend to be poorly indexed in standard bibliographic databases, information technology is important to consider alternative search methods for identifying relevant publications to be further screened for inclusion.
Strategy #2:
To identify books, volume chapters, and other not-journal publication types not thoroughly indexed in standard bibliographic databases, reviewers may cull to consult ane or more than of the post-obit less standard sources: Google Scholar, publisher web sites, or expert opinion.
In the example of the overview on sampling in qualitative research [eighteen], Google Scholar had ii advantages over other standard bibliographic databases: information technology indexes and returns records of books and book chapters probable to comprise guidance on qualitative research methods topics; and it has been validated as providing higher citation counts than ISI Web of Science (a producer of numerous bibliographic databases accessible through institutional subscription) for several non-biomedical disciplines including the social sciences where qualitative research methods are prominently used [19–21]. While we identified numerous useful publications by consulting experts, the author publication lists generated through Google Scholar searches were uniquely useful to identify more contempo editions of methods books identified by experts.
Searching without relevant metadata
Determining what publications to select for inclusion in the overview on sampling [xviii] could only rarely be achieved by reviewing the publication's metadata. This was because for the many books and other non-periodical blazon publications we identified as possibly relevant, the potential content of interest would be located in but a subsection of the publication. In this common scenario for reviews of the methods literature (equally opposed to methods overviews that include empirical report reports), reviewers will often be unable to employ standard title, abstract, and keyword database searching or screening equally a means for selecting publications.
Principle #3:
Considering that the presence of information about the topic of interest may not be indicated in the metadata for books and similar publication types, it is important to consider other ways of identifying potentially useful publications for further screening.
Strategy #iii:
One arroyo to identifying potentially useful books and similar publication types is to consider what classes of such publications (east.g., all methods manuals for a sure research approach) are likely to contain relevant content, so identify, call up, and review the full text of corresponding publications to make up one's mind whether they contain information on the topic of involvement.
In the example of the overview on sampling in qualitative research [eighteen], the topic of interest (sampling) was one of numerous topics covered in the general qualitative research methods manuals. Consequently, examples from this form of publications offset had to be identified for retrieval according to non-keyword-dependent criteria. Thus, all methods manuals inside the 3 research traditions reviewed (grounded theory, phenomenology, and case study) that might incorporate give-and-take of sampling were sought through Google Scholar and expert opinion, their full text obtained, and hand-searched for relevant content to make up one's mind eligibility. We used tables of contents and index sections of books to aid this mitt searching.
Purposefully selecting literature on conceptual grounds
A final consideration in methods overviews relates to the type of analysis used to generate the review findings. Unlike quantitative systematic reviews where reviewers aim for accurate or unbiased quantitative estimates—something that requires identifying and selecting the literature exhaustively to obtain all relevant data bachelor (i.e., a complete sample)—in methods overviews, reviewers must draw and translate the relevant literature in qualitative terms to achieve review objectives. In other words, the aim in methods overviews is to seek coverage of the qualitative concepts relevant to the methods topic at mitt. For example, in the overview of sampling in qualitative research [18], achieving review objectives entailed providing conceptual coverage of eight sampling-related topics that emerged as primal domains. The following principle recognizes that literature sampling should therefore support generating qualitative conceptual data as the input to analysis.
Principle #four:
Since the analytic findings of a systematic methods overview are generated through qualitative description and interpretation of the literature on a specified topic, pick of the literature should be guided by a purposeful strategy designed to achieve adequate conceptual coverage (i.due east., representing an appropriate caste of variation in relevant ideas) of the topic according to objectives of the review.
Strategy #4:
Ane strategy for choosing the purposeful approach to use in selecting the literature co-ordinate to the review objectives is to consider whether those objectives imply exploring concepts either at a broad overview level, in which case combining maximum variation choice with a strategy that limits yield (east.g., disquisitional case, politically important, or sampling for influence—described below) may be appropriate; or in depth, in which example purposeful approaches aimed at revealing innovative cases will likely be necessary.
In the methods overview on sampling, the unsaid scope was wide since nosotros set out to review publications on sampling across three divergent qualitative inquiry traditions—grounded theory, phenomenology, and case study—to facilitate making informative conceptual comparisons. Such an approach would be analogous to maximum variation sampling.
At the aforementioned time, the purpose of that review was to critically interrogate the clarity, consistency, and comprehensiveness of literature from these traditions that was "most probable to have widely influenced students' and researchers' ideas almost sampling" (p. 1774) [18]. In other words, nosotros explicitly set out to review and critique the most established and influential (and therefore ascendant) literature, since this represents a common basis of knowledge amongst students and researchers seeking agreement or practical guidance on sampling in qualitative research. To accomplish this objective, we purposefully sampled publications according to the criterion of influence, which we operationalized as how often an author or publication has been referenced in print or informal discourse. This second sampling approach likewise limited the literature nosotros needed to consider within our broad telescopic review to a manageable corporeality.
To operationalize this strategy of sampling for influence, we sought to identify both the near influential authors within a qualitative research tradition (all of whose citations were subsequently screened) and the most influential publications on the topic of interest by non-influential authors. This involved a flexible approach that combined multiple indicators of influence to avoid the dilemma that whatever single indicator might provide inadequate coverage. These indicators included bibliometric information (h-index for author influence [22]; number of cites for publication influence), expert stance, and cantankerous-references in the literature (i.e., snowball sampling). As a final option criterion, a publication was included only if information technology made an original contribution in terms of novel guidance regarding sampling or a related concept; thus, purely secondary sources were excluded. Publish or Perish software (Anne-Wil Harzing; available at http://www.harzing.com/resources/publish-or-perish) was used to generate bibliometric data via the Google Scholar database. Figure 1 illustrates how identification and selection in the methods overview on sampling was a multi-faceted and iterative procedure. The authors selected as influential, and the publications selected for inclusion or exclusion are listed in Additional file i (Matrices 1, 2a, 2b).
Literature identification and selection procedure used in the methods overview on sampling [18]
In summary, the strategies of seeking maximum variation and sampling for influence were employed in the sampling overview to meet the specific review objectives described. Reviewers volition demand to consider the total range of purposeful literature sampling approaches at their disposal in deciding what best matches the specific aims of their ain reviews. Suri [x] has recently retooled Patton'south well-known typology of purposeful sampling strategies (originally intended for primary enquiry) for application to literature synthesis, providing a useful resources in this respect.
Data brainchild
The purpose of data abstraction in rigorous literature reviews is to locate and record all data relevant to the topic of interest from the full text of included publications, making them bachelor for subsequent assay. Conventionally, a data abstraction class—consisting of numerous distinct conceptually defined fields to which corresponding information from the source publication is recorded—is adult and employed. There are several challenges, even so, to the processes of developing the abstraction class and abstracting the data itself when conducting methods overviews, which we address here. Some of these issues and their solutions may exist familiar to those who have conducted qualitative literature syntheses, which are similarly conceptual.
Iteratively defining conceptual information to abstract
In the overview on sampling [18], while we surveyed multiple sources beforehand to develop a listing of concepts relevant for abstraction (e.g., purposeful sampling strategies, saturation, sample size), there was no manner for the states to anticipate some concepts prior to encountering them in the review procedure. Indeed, in many cases, reviewers are unable to determine the complete set of methods-related concepts that will be the focus of the final review a priori without having systematically reviewed the publications to exist included. Thus, defining what information to abstract beforehand may not be feasible.
Principle #5:
Considering the potential impracticality of defining a consummate ready of relevant methods-related concepts from a body of literature 1 has not yet systematically read, selecting and defining fields for information abstraction must often be undertaken iteratively. Thus, concepts to exist bathetic can be expected to abound and alter as data abstraction proceeds.
Strategy #5:
Reviewers tin can develop an initial form or ready of concepts for abstraction purposes according to standard methods (east.grand., incorporating proficient feedback, airplane pilot testing) and remain attentive to the need to iteratively revise it as concepts are added or modified during the review. Reviewers should document revisions and render to re-abstract data from previously abstracted publications as the new data requirements are determined.
In the sampling overview [eighteen], we developed and maintained the brainchild course in Microsoft Word. Nosotros derived the initial set of abstraction fields from our own noesis of relevant sampling-related concepts, consultation with local experts, and reviewing a pilot sample of publications. Since the publications in this review included a large proportion of books, the abstraction process often began past flagging the broad sections within a publication containing topic-relevant information for detailed review to identify text to abstruse. When reviewing flagged text, the reviewer occasionally encountered an unanticipated concept meaning enough to warrant existence added as a new field to the abstraction class. For case, a field was added to capture how authors described the timing of sampling decisions, whether before (a priori) or later (ongoing) starting data collection, or whether this was unclear. In these cases, we systematically documented the modification to the form and returned to previously abstracted publications to abstract any data that might be relevant to the new field.
The logic of this strategy is analogous to the logic used in a form of inquiry synthesis called best fit framework synthesis (BFFS) [23–25]. In that method, reviewers initially code evidence using an a priori framework they have selected. When testify cannot be accommodated by the selected framework, reviewers then develop new themes or concepts from which they construct a new expanded framework. Both the strategy proposed and the BFFS approach to enquiry synthesis are notable for their rigorous and transparent means to adapt a final set up of concepts to the content under review.
Accounting for inconsistent terminology
An important complexity affecting the brainchild process in methods overviews is that the language used by authors to draw methods-related concepts can hands vary beyond publications. For example, authors from different qualitative research traditions often utilize different terms for similar methods-related concepts. Furthermore, as we constitute in the sampling overview [18], there may exist cases where no identifiable term, phrase, or label for a methods-related concept is used at all, and a description of information technology is given instead. This can make searching the text for relevant concepts based on keywords unreliable.
Principle #half-dozen:
Since accepted terms may not be used consistently to refer to methods concepts, it is necessary to rely on the definitions for concepts, rather than keywords, to identify relevant information in the publication to abstract.
Strategy #6:
An effective means to systematically place relevant information is to develop and iteratively adjust written definitions for central concepts (corresponding to brainchild fields) that are consequent with and every bit inclusive of as much of the literature reviewed as possible. Reviewers and then seek data that matches these definitions (rather than keywords) when scanning a publication for relevant data to abstruse.
In the abstraction process for the sampling overview [eighteen], nosotros noted the several concepts of interest to the review for which abstraction by keyword was particularly problematic due to inconsistent terminology beyond publications: sampling, purposeful sampling, sampling strategy, and saturation (for examples, encounter Additional file 1, Matrices 3a, 3b, 4). We iteratively developed definitions for these concepts by abstracting text from publications that either provided an explicit definition or from which an implicit definition could be derived, which was recorded in fields dedicated to the concept's definition. Using a method of constant comparing, we used text from definition fields to inform and alter a centrally maintained definition of the corresponding concept to optimize its fit and inclusiveness with the literature reviewed. Table one shows, equally an example, the final definition constructed in this fashion for ane of the central concepts of the review, qualitative sampling.
We practical iteratively developed definitions when making decisions nigh what specific text to abstract for an existing field, which allowed the states to abstract concept-relevant data even if no recognized keyword was used. For example, this was the case for the sampling-related concept, saturation, where the relevant text available for abstraction in one publication [26]—"to proceed to collect data until nada new was being observed or recorded, no thing how long that takes"—was not accompanied by whatever term or label whatsoever.
This comparative analytic strategy (and our arroyo to analysis more than broadly equally described in strategy #seven, below) is analogous to the process of reciprocal translation—a technique first introduced for meta-ethnography by Noblit and Hare [27] that has since been recognized as a common element in a variety of qualitative metasynthesis approaches [28]. Reciprocal translation, taken broadly, involves making sense of a study'south findings in terms of the findings of the other studies included in the review. In do, it has been operationalized in different ways. Melendez-Torres and colleagues adult a typology from their review of the metasynthesis literature, describing iv overlapping categories of specific operations undertaken in reciprocal translation: visual representation, fundamental paper integration, information reduction and thematic extraction, and line-by-line coding [28]. The approaches suggested in both strategies #vi and #seven, with their emphasis on constant comparison, appear to fall inside the line-by-line coding category.
Assay
Generating apparent and verifiable analytic interpretations
The assay in a systematic methods overview must support its more than general objective, which we suggested in a higher place is oftentimes to offer clarity and enhance collective understanding regarding a chosen methods topic. In our experience, this involves describing and interpreting the relevant literature in qualitative terms. Furthermore, any interpretative analysis required may entail reaching different levels of abstraction, depending on the more specific objectives of the review. For case, in the overview on sampling [18], we aimed to produce a comparative analysis of how multiple sampling-related topics were treated differently within and amidst different qualitative research traditions. To promote credibility of the review, however, not merely should one seek a qualitative analytic approach that facilitates reaching varying levels of abstraction but that approach must besides ensure that abstruse interpretations are supported and justified by the source data and not solely the production of the analyst's speculative thinking.
Principle #7:
Because the qualitative nature of the analysis required in systematic methods overviews, information technology is important to select an analytic method whose interpretations can be verified as being consistent with the literature selected, regardless of the level of abstraction reached.
Strategy #seven:
We suggest employing the constant comparative method of analysis [29] considering it supports developing and verifying analytic links to the source information throughout progressively interpretive or abstract levels. In applying this approach, we propose a rigorous arroyo, documenting how supportive quotes or references to the original texts are carried forward in the successive steps of analysis to let for easy verification.
The analytic arroyo used in the methods overview on sampling [eighteen] comprised four explicit steps, progressing in level of abstraction—data abstraction, matrices, narrative summaries, and final analytic conclusions (Fig. 2). While we have positioned information brainchild equally the second stage of the generic review procedure (prior to Assay), above, we also considered it as an initial stride of analysis in the sampling overview for several reasons. First, it involved a process of constant comparisons and iterative conclusion-making about the fields to add or ascertain during development and modification of the brainchild grade, through which we established the range of concepts to be addressed in the review. At the same time, brainchild involved continuous analytic decisions about what textual quotes (ranging in size from brusk phrases to numerous paragraphs) to record in the fields thus created. This constant comparative procedure was analogous to open up coding in which textual data from publications was compared to conceptual fields (equivalent to codes) or to other instances of data previously bathetic when constructing definitions to optimize their fit with the overall literature as described in strategy #6. Finally, in the information abstraction step, we also recorded our start interpretive thoughts in defended fields, providing initial material for the more abstract analytic steps.
Summary of progressive steps of assay used in the methods overview on sampling [18]
In the second step of the assay, we constructed topic-specific matrices, or tables, past copying relevant quotes from brainchild forms into the appropriate cells of matrices (for the consummate set of analytic matrices developed in the sampling review, encounter Boosted file i (matrices iii to 10)). Each matrix ranged from i to five pages; row headings, nested three-deep, identified the methodological tradition, author, and publication, respectively; and cavalcade headings identified the concepts, which corresponded to abstraction fields. Matrices thus immune us to make further comparisons across methodological traditions, and betwixt authors within a tradition. In the tertiary step of analysis, nosotros recorded our comparative observations as narrative summaries, in which we used illustrative quotes more sparingly. In the terminal footstep, we developed analytic conclusions based on the narrative summaries most the sampling-related concepts within each methodological tradition for which clarity, consistency, or comprehensiveness of the available guidance appeared to be lacking. Higher levels of analysis thus built logically from the lower levels, enabling us to easily verify analytic conclusions past tracing the support for claims past comparing the original text of publications reviewed.
Integrative versus interpretive methods overviews
The analytic production of systematic methods overviews is comparable to qualitative evidence syntheses, since both involve describing and interpreting the relevant literature in qualitative terms. Virtually qualitative synthesis approaches strive to produce new conceptual understandings that vary in level of estimation. Dixon-Forest and colleagues [30] elaborate on a useful distinction, originating from Noblit and Hare [27], betwixt integrative and interpretive reviews. Integrative reviews focus on summarizing bachelor primary data and involve using largely secure and well defined concepts to do so; definitions are used from an early on stage to specify categories for abstraction (or coding) of data, which in plow supports their aggregation; they do non seek equally their master focus to develop or specify new concepts, although they may accomplish some theoretical or interpretive functions. For interpretive reviews, meanwhile, the main focus is to develop new concepts and theories that integrate them, with the implication that the concepts developed get fully defined towards the end of the analysis. These two forms are non completely distinct, and "every integrative synthesis volition include elements of interpretation, and every interpretive synthesis volition include elements of assemblage of data" [30].
The instance methods overview on sampling [18] could exist classified equally predominantly integrative considering its primary goal was to amass influential authors' ideas on sampling-related concepts; there were too, however, elements of interpretive synthesis since it aimed to develop new ideas about where clarity in guidance on certain sampling-related topics is defective, and definitions for some concepts were flexible and not fixed until late in the review. We suggest that most systematic methods overviews will be classifiable equally predominantly integrative (aggregative). Nevertheless, more than highly interpretive methods overviews are also quite possible—for example, when the review objective is to provide a highly critical analysis for the purpose of generating new methodological guidance. In such cases, reviewers may demand to sample more than deeply (see strategy #4), specifically past selecting empirical research reports (i.east., to go beyond dominant or influential ideas in the methods literature) that are likely to characteristic innovations or instructive lessons in employing a given method.
Conclusions
In this newspaper, we have outlined tentative guidance in the class of vii principles and strategies on how to conduct systematic methods overviews, a review blazon in which methods-relevant literature is systematically analyzed with the aim of offer clarity and enhancing collective understanding regarding a specific methods topic. Our proposals include strategies for delimiting the set of publications to consider, searching across standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology, and generating credible and verifiable analytic interpretations. We hope the suggestions proposed volition exist useful to others undertaking reviews on methods topics in future.
As far as we are enlightened, this is the first published source of concrete guidance for conducting this blazon of review. It is important to annotation that our primary objective was to initiate methodological discussion by stimulating reflection on what rigorous methods for this type of review should look like, leaving the development of more complete guidance to future work. While derived from the experience of reviewing a single qualitative methods topic, we believe the principles and strategies provided are generalizable to overviews of both qualitative and quantitative methods topics alike. However, it is expected that boosted challenges and insights for conducting such reviews accept still to be defined. Thus, we suggest that next steps for developing more definitive guidance should involve an attempt to collect and integrate other reviewers' perspectives and experiences in conducting systematic methods overviews on a broad range of qualitative and quantitative methods topics. Formalized guidance and standards would improve the quality of time to come methods overviews, something we believe has important implications for advancing qualitative and quantitative methodology. When undertaken to a loftier standard, rigorous critical evaluations of the available methods guidance have significant potential to brand implicit controversies explicit, and improve the clarity and precision of our understandings of problematic qualitative or quantitative methods problems.
A review procedure central to nearly types of rigorous reviews of empirical studies, which we did not explicitly address in a divide review step above, is quality appraisement. The reason we have not treated this as a separate pace stems from the different objectives of the primary publications included in overviews of the methods literature (i.e., providing methodological guidance) compared to the primary publications included in the other established review types (i.e., reporting findings from single empirical studies). This is not to say that appraising quality of the methods literature is not an important concern for systematic methods overviews. Rather, appraisal is much more than integral to (and hard to carve up from) the analysis step, in which we advocate appraising clarity, consistency, and comprehensiveness—the quality appraisal criteria that nosotros advise are appropriate for the methods literature. As a second of import deviation regarding appraisement, we currently advocate appraising the aforementioned aspects at the level of the literature in aggregate rather than at the level of private publications. One reason for this is that methods guidance from private publications generally builds on previous literature, and thus we feel that ahistorical judgments about comprehensiveness of unmarried publications lack relevance and utility. Additionally, while dissimilar methods authors may limited themselves less clearly than others, their guidance tin nevertheless exist highly influential and useful, and should therefore not exist downgraded or ignored based on considerations of clarity—which raises questions near the culling uses that quality appraisals of individual publications might have. Finally, legitimate variability in the perspectives that methods authors wish to emphasize, and the levels of generality at which they write nearly methods, makes critiquing individual publications based on the benchmark of clarity a complex and potentially problematic effort that is beyond the scope of this paper to accost. By appraising the electric current state of the literature at a holistic level, reviewers stand to identify important gaps in understanding that represent valuable opportunities for further methodological evolution.
To summarize, the principles and strategies provided here may be useful to those seeking to undertake their ain systematic methods overview. Additional work is needed, notwithstanding, to establish guidance that is comprehensive by comparison the experiences from conducting a variety of methods overviews on a range of methods topics. Efforts that further accelerate standards for systematic methods overviews take the potential to promote high-quality critical evaluations that produce conceptually clear and unified understandings of problematic methods topics, thereby accelerating the accelerate of research methodology.
References
-
Hutton JL, Ashcroft R. What does "systematic" mean for reviews of methods? In: Black Due north, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods: a guide to best practice. London: BMJ Publishing Group; 1998. p. 249–54.
-
Cochrane handbook for systematic reviews of interventions. In. Edited past Higgins JPT, Green S, Version 5.i.0 edn: The Cochrane Collaboration; 2011.
-
Centre for Reviews and Dissemination: Systematic reviews: CRD's guidance for undertaking reviews in health care. York: Centre for Reviews and Dissemination; 2009.
-
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA argument for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700–0.
-
Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.
-
Kastner K, Tricco Ac, Soobiah C, Lillie E, Perrier L, Horsley T, Welch 5, Cogo Eastward, Antony J, Straus SE. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12(1):i–ane.
-
Berth A, Noyes J, Flemming K, Gerhardus A. Guidance on choosing qualitative testify synthesis methods for apply in health engineering science assessments of complex interventions. In: Integrate-HTA. 2016.
-
Booth A, Sutton A, Papaioannou D. Systematic approaches to successful literature review. second ed. London: Sage; 2016.
-
Hannes Grand, Lockwood C. Synthesizing qualitative research: choosing the correct arroyo. Chichester: Wiley-Blackwell; 2012.
-
Suri H. Towards methodologically inclusive research syntheses: expanding possibilities. New York: Routledge; 2014.
-
Campbell M, Egan M, Lorenc T, Bond Fifty, Popham F, Fenton C, Benzeval M. Considering methodological options for reviews of theory: illustrated past a review of theories linking income and health. Syst Rev. 2014;3(1):1–11.
-
Cohen DJ, Crabtree BF. Evaluative criteria for qualitative inquiry in wellness intendance: controversies and recommendations. Ann Fam Med. 2008;half dozen(4):331–9.
-
Tong A, Sainsbury P, Craig J. Consolidated criteria for reportingqualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.
-
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;vii(2):e1000217.
-
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.
-
Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365(9465):1159–62.
-
Alshurafa Thou, Briel M, Akl EA, Haines T, Moayyedi P, Gentles SJ, Rios 50, Tran C, Bhatnagar N, Lamontagne F, et al. Inconsistent definitions for intention-to-treat in relation to missing outcome information: systematic review of the methods literature. PLoS 1. 2012;7(11):e49163.
-
Gentles SJ, Charles C, Ploeg J, McKibbon KA. Sampling in qualitative research: insights from an overview of the methods literature. Qual Rep. 2015;20(xi):1772–89.
-
Harzing A-W, Alakangas South. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics. 2016;106(2):787–804.
-
Harzing A-WK, van der Wal R. Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit. 2008;8(i):61–73.
-
Kousha One thousand, Thelwall M. Google Scholar citations and Google Web/URL citations: a multi‐discipline exploratory analysis. J Assoc Inf Sci Technol. 2007;58(7):1055–65.
-
Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A. 2005;102(46):16569–72.
-
Booth A, Carroll C. How to build upwards the actionable knowledge base: the role of 'best fit' framework synthesis for studies of improvement in healthcare. BMJ Quality Safety. 2015;24(11):700–8.
-
Carroll C, Booth A, Leaviss J, Rick J. "Best fit" framework synthesis: refining the method. BMC Med Res Methodol. 2013;13(ane):37.
-
Carroll C, Booth A, Cooper K. A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;xi(one):29.
-
Cohen MZ, Kahn DL, Steeves DL. Hermeneutic phenomenological research: a applied guide for nurse researchers. Thousand Oaks: Sage; 2000.
-
Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: Sage; 1988.
-
Melendez-Torres GJ, Grant S, Bonell C. A systematic review and critical appraisal of qualitative metasynthetic practice in public health to develop a taxonomy of operations of reciprocal translation. Res Synthesis Methods. 2015;6(4):357–71.
-
Glaser BG, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.
-
Dixon-Wood M, Agarwal South, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. In: United kingdom of great britain and northern ireland National Wellness Service. 2004. p. 1–44.
Acknowledgements
Not applicable.
Funding
There was no funding for this work.
Availability of data and materials
The systematic methods overview used as a worked case in this article (Gentles SJ, Charles C, Ploeg J, McKibbon KA: Sampling in qualitative research: insights from an overview of the methods literature. The Qual Rep 2015, xx(11):1772-1789) is available from http://nsuworks.nova.edu/tqr/vol20/iss11/5.
Authors' contributions
SJG wrote the first draft of this commodity, with CC contributing to drafting. All authors contributed to revising the manuscript. All authors except CC (deceased) canonical the terminal draft. SJG, CC, KAB, and JP were involved in developing methods for the systematic methods overview on sampling.
Authors' data
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Non applicative.
Ethics approval and consent to participate
Not applicable.
Author information
Affiliations
Corresponding author
Additional information
Cathy Charles is deceased
Additional file
Rights and permissions
Open Admission This article is distributed nether the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in whatever medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made bachelor in this article, unless otherwise stated.
Reprints and Permissions
About this article
Cite this article
Gentles, S.J., Charles, C., Nicholas, D.B. et al. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative enquiry. Syst Rev v, 172 (2016). https://doi.org/10.1186/s13643-016-0343-0
-
Received:
-
Accepted:
-
Published:
-
DOI : https://doi.org/10.1186/s13643-016-0343-0
Keywords
- Systematic review
- Literature choice
- Inquiry methods
- Inquiry methodology
- Overview of methods
- Systematic methods overview
- Review methods
Source: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0343-0
Post a Comment for "Extensive Review of Literature Qualitative or Quantitative Research"