ORIGINAL ARTICLE
Big data analytics capabilities: a systematic literature review and research agenda
Patrick Mikalef1 • Ilias O. Pappas1 • John Krogstie1 •
Michail Giannakos1
Received: 15 November 2016 / Revised: 3 July 2017 / Accepted: 12 July 2017 /
Published online: 15 July 2017
� Springer-Verlag GmbH Germany 2017
Abstract With big data growing rapidly in importance over the past few years, academics and practitioners have been considering the means through which they
can incorporate the shifts these technologies bring into their competitive strategies.
To date, emphasis has been on the technical aspects of big data, with limited
attention paid to the organizational changes they entail and how they should be
leveraged strategically. As with any novel technology, it is important to understand
the mechanisms and processes through which big data can add business value to
companies, and to have a clear picture of the different elements and their interde-
pendencies. To this end, the present paper aims to provide a systematic literature
review that can help to explain the mechanisms through which big data analytics
(BDA) lead to competitive performance gains. The research framework is grounded
on past empirical work on IT business value research, and builds on the resource-
based view and dynamic capabilities view of the firm. By identifying the main areas
of focus for BDA and explaining the mechanisms through which they should be
leveraged, this paper attempts to add to literature on how big data should be
examined as a source of competitive advantage. To this end, we identify gaps in the
extant literature and propose six future research themes.
Keywords Big data � Dynamic capabilities � Resource-based view � Competitive performance � IT strategy
& Patrick Mikalef [email protected]
1 Norwegian University of Science and Technology, Trondheim, Norway
123
Inf Syst E-Bus Manage (2018) 16:547–578
https://doi.org/10.1007/s10257-017-0362-y
1 Introduction
The application of big data in driving organizational decision making has attracted
much attention over the past few years. A growing number of firms are focusing
their investments on big data analytics (BDA) with the aim of deriving important
insights that can ultimately provide them with a competitive edge (Constantiou and
Kallinikos 2015). The need to leverage the full potential of the rapidly expanding
data volume, velocity, and variety has seen a significant evolution of techniques and
technologies for data storage, analysis, and visualization. However, there has been
considerably less research attention on how organizations need to change in order to
embrace these technological innovations, as well as on the business shifts they entail
(McAfee et al. 2012). Despite the hype surrounding big data, the issue of examining
whether, and under what conditions, big data investments produce business value,
remains underexplored, severely hampering their business and strategic potential
(McAfee et al. 2012). Most studies to date have primarily focused on infrastructure,
intelligence, and analytics tools, while other related resources, such as human skills
and knowledge, have been largely disregarded. Furthermore, orchestration of these
resources, the socio-technological developments that they precipitate, as well as
how they should be incorporated into strategy and operations thinking, remains an
underdeveloped area of research (Gupta and George 2016).
Over the past few years, several research commentaries have stressed the
importance of delving into the whole spectrum of aspects that surround BDA
(Constantiou and Kallinikos 2015; Markus 2015). Nevertheless, exploratory
empirical literature on the topic is still quite scarce (Gupta and George 2016;
Wamba et al. 2017). Past literature reviews on the broader information systems (IS)
domain have demonstrated that there are multiple aspects that should be considered
when examining the business potential of IT investments (Schryen 2013).
Furthermore, the particularities of each technological development need to be
thoroughly examined in order to fully capture the interdependencies that develop
between them, and how they produce value at a firm level. Past literature on IT
business value has predominantly used the notion of IT capabilities to refer to the
broader context of technology within firms, and the overall proficiency in leveraging
and mobilizing the different resources and capabilities (Bharadwaj 2000). It is
therefore important to identify and explore the domain-specific aspects that are
relevant to BDA within the business context (Kamioka and Tapanainen 2014).
While there is a growing stream of literature on the business potential of BDA,
there is still limited work grounded on established theories used in the IT-business
value domain (Gupta and George 2016). The lack of empirical work in this direction
significantly hinders research concerning the value of BDA, and leaves practitioners
in unchartered territories when faced with implementing such initiatives in their
firms. Hence, in order to derive meaningful theoretical and practical implications, as
well as to identify important areas of future research, it is critical to understand how
the core artifacts pertinent to BDA are shaped, and how they lead to business value
(Constantiou and Kallinikos 2015). Therefore, we employ a systematic literature
review grounded in the established resource-based view (RBV) of the firm, as well
548 P. Mikalef et al.
123
as the emerging dynamic capabilities view (DCV). We select these theoretical
groundings since the former provides a solid foundation upon which all relevant
resources can be identified and evaluated towards their importance, while the latter
enables examination of the organizational capabilities towards which these
resources should be directed in order to achieve competitive performance gains
(Mikalef et al. 2016a, b). As such, the DCV exerts complementarities in relation to
the RBV by providing an explanation of the rent-yielding properties of organiza-
tional capabilities that can be leveraged by means of BDA (Makadok 2001). Our
theoretical framework that guides the systematic literature review uncovers some
initial findings on the value of BDA, while also providing a roadmap on several
promising research streams.
The rest of the paper is structured as follows. In Sect. 2, we describe the research
methodology used to conduct the systematic literature review, and outline the main
steps followed. Next, in Sect. 3, we distinguish between the concepts of big data,
BDA, and BDA capability, and present some definitions as described in literature
for each. In Sect. 4, we proceed to describe the main theoretical foundations upon
which we build on and develop the proposed research framework. We then
summarize existing work on the business value of BDA according to the identified
themes. In Sect. 5, we outline a series of areas that are currently under-researched
and propose appropriate theoretical stances that could be utilized in their
examination. In closing, Sect. 6 presents some concluding remarks on the area of
BDA and their application to the strategic domain.
2 Research methodology
Following the established method of a systematic literature review (Kitchenham
2004, 2007; Kitchenham et al. 2009), we undertook the review in distinct stages.
First, we developed the review protocol. Second, we identified the inclusion and
exclusion criteria for relevant publications. Third, we performed an in-depth search
for studies, followed by critical appraisal, data extraction and a synthesis of past
findings. The next sub-sections describe in detail the previously mentioned stages
(Fig. 1).
2.1 Protocol development
The first step of the systematic literature review was to develop a protocol for the
next steps. In accordance with the guideline, procedures, and policies of the
Cochrance Handbook for Systematic Reviews of Intervention (Higgins and Green
2008), the protocol established the main research question that guided the selection
of papers, the search strategy, inclusion and quality criteria, as well as the method of
synthesis. The review process was driven by the following research question: What
are the definitional aspects, unique characteristics, challenges, organizational
transformations, and business value associated with big data? By focusing on these
elements of the research question, the subject areas and relevant publications and
materials were identified.
Big data analytics capabilities: a systematic literature… 549
123
2.2 Inclusion and exclusion criteria
Due to the importance of the selection phase in determining the overall validity of
the literature review, a number of inclusion and exclusion criteria were applied.
Studies were eligible for inclusion if they were focused on the topic of how big data
can provide business value. Publications were selected from 2010 onwards, since
that is when the term gained momentum in the academic and business communities.
The systematic review included research papers published in academic outlets, such
as journal articles and conference proceedings, as well as reports targeted at
business executives and a broader audience, such as scientific magazines. In-
progress research and dissertations were excluded from this review, as were studies
that were not written in English. Finally, given that our focus was on the business
transformation that big data entails, along with performance outcomes, we included
quantitative, qualitative, and case studies. Since the topic of interest is of an
interdisciplinary nature, a diversity of epistemological approaches was opted for.
2.3 Data sources and search strategy
The search strategy started by forming search strings that were then combined to
form keywords. In addition, during the search we employed wildcard symbols in
Fig. 1 Stages of the study selection process
550 P. Mikalef et al.
123
order to reduce the number of search strings. Combinations of two sets of keywords
were used, with the first term being ‘big data,’ and the second term being one of 12,
which were reviewed by a panel of five experts. These search terms included:
analytics capability, competitive performance, firm performance, organizational
performance, dynamic capabilities, resource-based view, human skills, managerial
skills, analytics ecosystems, data scientist, competencies, and resource management.
Keywords were searched within the title, abstract, and keyword sections of the
manuscripts. The search strategy included electronic databases such as Scopus,
Business Source Complete, Emerald, Taylor & Francis, Springer, Web of
Knowledge, ABI/inform Complete, IEEE Xplore, and the Association of Informa-
tion Systems (AIS) library. To further complement our search, we applied the search
terms in the search engine Google Scholar, as well as the AIS basket of eight
journals.
The search was initiated on September 5, 2016 and ended on February 26, 2017.
At stage 1, 459 papers were identified and entered into the reference manager
EndNote. At stage 2, all authors went through the titles of the studies of stage 1 in
order to determine their relevance to the systematic review. At this stage, studies
that were clearly not about the business aspects of big data were excluded,
independently of whether they were empirical. In addition, articles that were
focused on big data for public administration were not included in the next stage.
The number of retained articles after the abovementioned process was 228. In the
third stage, all remaining articles were examined in terms of their abstracts and their
focus in relation to the research question we had defined. However, some abstracts
were of varying quality, some were lacking information about the content of the
article, while others had an apparent disconnect with their title and did not fit our
focus. At this stage, each papers’ abstract was reviewed independently by each
author. From the 228 abstracts assessed, 101 were omitted, leaving 127 papers to be
further analyzed.
2.4 Quality assessment
Each of the 127 papers that remained after stage 3 was assessed independently by
the authors in terms of several quality criteria. Studies were examined in terms of
scientific rigor, so that appropriate research methods had been applied; credibility,
to assess whether findings were well presented; and relevance, which was assessed
based on whether findings were useful for companies engaging in big data projects,
as well as the academic community. Taken together, these criteria provided a
measure of the extent to which a publication would make a valuable contribution to
the review. At this stage another 43 papers were excluded, leaving 84 papers for
data extraction and synthesis. These papers were then coded according to their area
of focus, allowing a categorization to be constructed. The derived categories were a
result of identifying the main research areas that papers aimed to contribute towards.
By categorizing papers, we were able to extract the details needed to answer each of
the posed research questions.
Big data analytics capabilities: a systematic literature… 551
123
2.5 Data extraction and synthesis of findings
In order to synthesize findings and categorize studies based on their scope, an
analysis of the different research streams was performed. The first step was to
identify the main concepts from each study, using the authors’ original terms. The
key concepts were then organized in a spreadsheet in order to enable comparison
across studies and translation of findings into higher-order interpretations. An
analysis was conducted based on the following areas of focus: organizational
performance outcomes of big data, human skills and knowledge, tangible and
intangible resources, team orchestration and project management, adoption and
diffusion of big data initiatives, governance in big data projects, as well as ethical
and moral issues related to big data within the business domain. For empirical
studies, the the authors also recorded the type of study conducted (e.g. qualitative,
quantitative, case study), the sample size, the instruments used (e.g. surveys,
interviews, observations), as well as contextual factors surrounding the study (e.g.
industry, country, firm size). Constant consensus meetings of all researchers
established the data extraction stage and the categorization of publications. The
remaining 84 papers were analyzed in detail in accordance with the coding scheme,
and relevant data were extracted, analyzed, and synthesized.
3 Defining big data in the business context
Big data is becoming an emerging topic of interest in IS, computer and information
sciences, management, and social sciences (Constantiou and Kallinikos 2015). This
phenomenon is largely attributed to the widespread adoption of social media,
mobile devices and sensors, integrated IS, and artifacts related to the Internet of
Things. The surging interest in big data is also reflected in the academic literature,
which spans multiple disciplinary domains (Chen et al. 2016). While the different
epistemological domains provide an alternative perspective on the notion of big
data, the definitions and key concepts put forth by each differ significantly (Wamba
et al. 2015). As such, the first step of the systematic literature review is to identify
the key concepts and develop integrative definitions of each. Notions such as big
data, BDA, and BDA capability are often used interchangeably in the literature.
However, their theoretical underpinnings reflect a different perspective in how they
are perceived and measured (Cao and Duan 2014a). Therefore, it is imperative to
clearly define the meaning of these concepts, and that aspects they encompass.
3.1 Big data
As a starting point, we provide an overview of how big data have been defined in
past studies, as well as what attributes are integral to the concept. Several definitions
of big data have been put forth to date in attempts to distinguish the phenomenon of
big data from conventional data-driven or business analytics approaches (Table 1).
Some scholars focus on the origin of the data, emphasizing the various channels
from which they are collected, such as enterprise IS, customer transactions,
552 P. Mikalef et al.
123
Table 1 Sample definitions of big data
Author(s) and date Definition
Russom (2011) Big data involves the data storage, management, analysis, and visualization
of very large and complex datasets
White (2011) Big data involves more than simply the ability to handle large volumes of
data; instead, it represents a wide range of new analytical technologies and
business possibilities. These new systems handle a wide variety of data,
from sensor data to Web and social media data, improved analytical
capabilities, operational business intelligence that improves business agility
by enabling automated real-time actions and intraday decision making,
faster hardware and cloud computing including on-demand software-as-a
service. Supporting big data involves combining these technologies to
enable new solutions that can bring significant benefits to the business
Beyer and Laney (2012) Big data: high-volume, high-velocity, and/or high-variety information assets
that require new forms of processing to enable enhanced decision making,
insight discovery, and process optimization
McAfee et al. (2012) Big data, like analytics before it, seeks to glean intelligence from data and
translate that into business advantage. However, there are three key
differences: Velocity, variety, volume
Gantz and Reinsel (2012) Big data focuses on three main characteristics: the data itself, the analytics of
the data, and presentation of the results of the analytics that allow the
creation of business value in terms of new products or services
Boyd and Crawford
(2012)
Big data: a cultural, technological, and scholarly phenomenon that rests on
the interplay of (1) Technology: maximizing computation power and
algorithmic accuracy to gather, analyze, link, and compare large datasets.
(2) Analysis: drawing on large datasets to identify patterns in order to make
economic, social, technical, and legal claims. (3) Mythology: the
widespread belief that large datasets offer a higher form of intelligence and
knowledge that can generate insights that were previously impossible, with
the aura of truth, objectivity, and accuracy
Schroeck et al. (2012) Big data is a combination of volume, variety, velocity and veracity that
creates an opportunity for organizations to gain competitive advantage in
today’s digitized marketplace
Bharadwaj et al. (2013) Big data refers to datasets with sizes beyond the ability of common software
tools to capture, curate, manage, and process the data within a specified
elapsed time
Kamioka and Tapanainen
(2014)
Big data is large-scale data with various sources and structures that cannot be
processed by conventional methods and that is intended for organizational
or societal problem solving
Bekmamedova and
Shanks (2014)
Big data involves the data storage, management, analysis, and visualization
of very large and complex datasets. It focuses on new data-management
techniques that supersede traditional relational systems, and are better
suited to the management of large volumes of social media data
Davis (2014) Big data consists of expansive collections of data (large volumes) that are
updated quickly and frequently (high velocity) and that exhibit a huge
range of different formats and content (wide variety)
Sun et al. (2015) Big data: the data-sets from heterogeneous and autonomous resources, with
diversity in dimensions, complex and dynamic relationships, by size that is
beyond the capacity of conventional processes or tools to effectively
capture, store, manage, analyze, and exploit them
Big data analytics capabilities: a systematic literature… 553
123
machines or sensors, social media, cell phones or other networked devices, news
and network content, as well as GPS signals (Chen et al. 2016; Opresnik and Taisch
2015). The majority of scholars emphasize the ‘‘three Vs’’ that characterize big data:
volume, velocity, and variety (McAfee et al. 2012; Davis 2014; Sun et al. 2015).
Volume refers to the sheer size of the dataset due to the aggregation of a large
number of variables and an even larger set of observations for each variable.
(George et al. 2016). In addition, many definitions highlight the growing rate at
which the quantity of data increases, commonly expressed in petabytes or exabytes,
used by decision makers to aid strategic decisions (Akter et al. 2016a). Velocity
reflects the speed at which these data are collected, updated, and analyzed, as well as
the rate at which their value becomes obsolete (Davis 2014; George et al. 2016).
The ‘newness’ of data that decision makers are able to collect, as well as the
capacity to analyze these data-streams, is an important factor when it comes to
improving business agility and enabling real-time actions and intraday decision
making (White 2011; Boyd and Crawford 2012). Variety refers to the plurality of
structured and unstructured data sources, which, amongst others, include text, audio,
images, video, networks, and graphics (Constantiou and Kallinikos 2015; George
et al. 2016). While there are no universal benchmarks for defining the volume,
velocity, and variety of big data, the defining limits are contingent upon size, sector,
and location of the firm, and are subject to changing limits over time (Gandomi and
Haider 2015).
Adding to the existing body of definitions, several scholars have included
different aspects of big data in their conceptualizations (Table 2). For instance, a
commonly acknowledged aspect of big data is its veracity (Akter et al. 2016a, b;
Table 1 continued
Author(s) and date Definition
Opresnik and Taisch
(2015)
Big data typically refers to the following types of data: (1) traditional
enterprise data, (2) machine-generated/sensor data (e.g. weblogs, smart
meters, manufacturing sensors, equipment logs), and (3) social data
Constantiou and
Kallinikos (2015)
Big data often represents miscellaneous records of the whereabouts of large
and shifting online crowds. It is frequently agnostic, in the sense of being
produced for generic purposes or purposes different from those sought by
big data crunching. It is based on varying formats and modes of
communication (e.g. text, image, and sound), raising severe problems of
semiotic translation and meaning compatibility. Big data is commonly
deployed to refer to large data volumes generated and made available on
the Internet and the current digital media ecosystems
Akter et al. (2016a) Big data is defined in terms of five ‘Vs:’ volume, velocity, variety, veracity,
and value. ‘Volume’ refers to the quantities of big data, which are
increasing exponentially. ‘Velocity’ is the speed of data collection,
processing and analyzing in the real time. ‘Variety’ refers to the different
types of data collected in big data environments. ‘Veracity’ represents the
reliability of data sources. Finally, ‘value’ represents the transactional,
strategic, and informational benefits of big data
Abbasi et al. (2016) Big data differs from ‘regular’ data along four dimensions, or ‘4 Vs’—
volume, velocity, variety, and veracity
554 P. Mikalef et al.
123
Abbasi et al. 2016). Veracity refers to the degree to which big data is trusted,
authentic, and protected from unauthorized access and modification (Demchenko
et al. 2013). Analyzing high-quality and reliable data is imperative in enabling
management to make cognizant decisions and derive business value (Akter et al.
2016b). Hence, big data used for business decisions should be authenticated and
have passed through strict quality-compliance procedures before being analyzed
(Dong and Srivastava 2013; Gandomi and Haider 2015). This vast amount of data is
argued to be an important enabler of creating value for organizations (Gandomi and
Haider 2015). Oracle introduced value as a defining aspect of big data. According to
Oracle’s (2012) definition, big data are frequently characterized by low value
density, meaning that the value of the processed data is proportionately low
compared to its volume. Seddon and Currie (2017) included two additional
dimensions in the definition of big data: variability and visualization. Variability
refers to the dynamic opportunities that are available by interpreting big data, while
visualization has to do with the representation of data in meaningful ways through
artificial intelligence methods that generate models (Seddon and Currie 2017).
3.2 Big data analytics
Some definitions of big data focus solely on the data and their defining
characteristics (Davis 2014; Akter et al. 2016a, b; Abbasi et al. 2016); others
extend and include the analytical procedures, tools, and techniques that are
Table 2 Defining characteristics of big data
Attribute Definition
Volume Volume represents the sheer size of the dataset due to the aggregation of a large number
of variables and an even larger set of observations for each variable. (George et al.
2016)
Velocity Velocity reflects the speed at which data are collected and analyzed, whether in real time
or near real time from sensors, sales transactions, social media posts, and sentiment data
for breaking news and social trends. (George et al. 2016)
Variety Variety in big data comes from the plurality of structured and unstructured data sources
such as text, videos, networks, and graphics among others. (George et al. 2016)
Veracity Veracity ensures that the data used are trusted, authentic, and protected from
unauthorized access and modification. (Demchenko et al. 2013)
Value Value represents the extent to which big data generates economically worthy insights
and/or benefits through extraction and transformation. (Wamba et al. 2015)
Variability Variability concerns how insight from media constantly changes as the same information
is interpreted in a different way, or new feeds from other sources help to shape a
different outcome. (Seddon and Currie 2017)
Visualization Visualization can be described as interpreting the patterns and trends that are present in
the data. (Seddon and Currie 2017)
3Vs: volume, velocity, variety (Chen and Zhang 2014)
4Vs: volume, velocity, variety, veracity (Zikopoulos and Eaton 2011; Schroeck et al. 2012; Abbasi et al.
2016)
5Vs: volume, velocity, variety, veracity, value (Oracle 2012; Sharda et al. 2013)
7Vs: volume, velocity, variety, veracity, value variability, visualization (Seddon and Currie 2017)
Big data analytics capabilities: a systematic literature… 555
123
employed (Russom 2011; Bharadwaj et al. 2013); while some even go on to
describe the type of impact that the analysis and presentation of big data can yield in
terms of business value (White 2011; Beyer and Laney 2012; Schroeck et al. 2012;
De Mauro et al. 2015). This point is made very clear by the definition provided by
Gantz and Reinsel (2012), who state that BDA revolve around three main
characteristics: the data itself, the analytics applied to the data, and the presentation
of results in a way that allows the creation of business value. In this definition, the
process of analyzing the data is outlined without linking it to any tangible or
intangible business outcome. George et al. (2016) posit that big data refers to large
and varied data that can be collected and managed, whereas data science develops
models that capture, visualize, and analyze the underlying patterns in the data. To
make this distinction more apparent, some scholars use the term BDA to emphasize
the process and tools used in order to extract insights from big data. In essence,
BDA encompasses not only the entity upon which analysis in performed—i.e. the
data—but also elements of tools, infrastructure, and means of visualizing and
presenting insight. This distinction is quite eloquently put in the definitions of Kwon
et al. (2014), and Lamba and Dubey (2015). Nevertheless, while the definitions of
BDA encompass a wider spectrum of elements critical to the success of big data,
they do not include the organizational resources that are required to transform big
data into actionable insight. Becoming a data-driven organization is a complex and
multifaceted task, and necessitates attention at multiple levels from managers. To
address the transition to a data-driven era and provide practitioners with guidelines
on how to deploy their big data initiatives, scholars have begun utilizing the term
‘BDA capability’ to reference a company’s proficiency in leveraging big data to
gain strategic and operational insight (Table 3).
Table 3 Sample definitions of big data analytics
Authors and date Definition
Loebbecke and Picot
(2015)
Big data analytics: a means to analyze and interpret any kind of digital
information. Technical and analytical advancements in BDA, which—in large
part—determine the functional scope of today’s digital products and services,
are crucial for the development of sophisticated artificial intelligence,
cognitive computing capabilities, and business intelligence
Kwon et al. (2014) Big data analytics: technologies (e.g. database and data mining tools) and
techniques (e.g. analytical methods) that a company can employ to analyze
large-scale, complex data for various applications intended to augment firm
performance in various dimensions
Ghasemaghaei et al.
(2015)
Big data analytics, defined as tools and processes often applied to large and
disperse datasets for obtaining meaningful insights, has received much
attention in IS research given its capacity to improve organizational
performance
Lamba and Dubey
(2015)
Big data analytics is defined as the application of multiple analytic methods that
address the diversity of big data to provide actionable descriptive, predictive,
and prescriptive results
Müller et al. (2016) Big data analytics: the statistical modeling of large, diverse, and dynamic
datasets of user-generated content and digital traces
556 P. Mikalef et al.
123
3.3 Big data analytics capability
Despite the limited published research on big data, some studies have focused on the
challenges that companies face during the implementation of big data projects (Gupta
and George 2016; Vidgen et al. 2017). Particularly within the IS domain, researchers
recognize that the success of big data projects is not only a result of the data and the
analytical tools and processes, but includes a broader range of aspects (Garmaki et al.
2016). To address this issue, the notion of BDA capability has been proposed, which
is broadly defined as the ability of a firm to provide insights using data management,
infrastructure, and talent to transform business into a competitive force (Kiron et al.
2014; Akter et al. 2016a). Research in this area focuses on strategy-driven BDA
capabilities, and the mechanisms through which competitive performance gains are
realized (LaValle et al. 2011). Some definitions of BDA capabilities focus on the
processes that must be put in place in order to leverage big data (Cao and Duan
2014b; Olszak 2014), while others emphasize the investment of necessary resources
and their alignment with strategy (Xu and Kim 2014). In essence, the notion of BDA
capability extends the view of big data to include all related organizational resources
that are important in leveraging big data to their full strategic potential (Table 4).
Table 4 Sample definitions of big data analytics capability
Author(s) and date Definition
Davenport and Harris
(2007)
BDA capability is defined as the distinctive capability of firms in setting the
optimal price, detecting quality problems, deciding the lowest possible level
of inventory, or identifying loyal and profitable customers in big data
environments
Cao and Duan (2014a) Information processing capabilities: an organization’s capacity to capture,
integrate, and analyze big data, and utilize insights derived from that big data
to make informed decisions that generate real business value
Xu and Kim (2014) Business intelligence capabilities: a combination of a set of sub-capabilities.
Derived from IT capabilities, we define business intelligence capabilities from
the perspectives of infrastructures, skills, execution, and relationship
Olszak (2014) Dynamic business intelligence capability is the ability of an organization to
integrate, build, and reconfigure the information resources, as well as business
processes, to address rapidly changing environments
Kung et al. (2015) Big data competence: a firm’s ability to acquire, store, process, and analyze
large amounts of data in various forms, and deliver information to users that
allows organizations to extract value from big data in a timely fashion
Big data resources are defined as a combination of complementary IT resources
relevant to the utilization of big data to enhance firm performance
Garmaki et al. (2016) The BDA capability entails a firm’s ability to mobilize and deploy BDA
resources effectively, utilize BDA resources, and align BDA planning with
firm strategy to gain competitive advantage and enhance firm performance
Shuradze and Wagner
(2016)
A data analytics capability can be defined as an organization’s ability to
mobilize and deploy data analytics-related resources in combination with
marketing resources and capabilities, which constitutes an innovative IT
capability that can improve firm performance
Gupta and George
(2016)
BDA capability is defined as a firm’s ability to assemble, integrate, and deploy
its big data-specific resources
Big data analytics capabilities: a systematic literature… 557
123
To date, there is limited empirical research building on the notion of BDA
capability. Most studies are based on anecdotal evidence, particularly in relation to
the impact of a firm’s BDA capability on performance (Agarwal and Dhar 2014;
Akter et al. 2016a). Furthermore, there are diverging views about what constitutes
BDA capability, since different theoretical lenses are often employed. In this regard,
the purpose of the following section is to provide a theoretically driven synthesis of
past studies concerning the aspects that are important in order to develop BDA
capability. Thus, we seek to distinguish between the notion of developing a BDA
capability and leveraging the competence of a firm to enable or strengthen certain
organizational capabilities by means of BDA. We then discuss how the former is a
prerequisite for the latter, yet the existence of a BDA capability does not
automatically mean that the leveraged competence is actualized.
4 Toward the development of a big data analytics capability
4.1 Resource based theory
Developing and sustaining competitive advantage is the cornerstone of strategic
management literature, which draws on a number of interwoven yet distinct
elements and notions (Wernerfelt 1984; Amit and Schoemaker 1993). Resource-
based theory (RBT) has been widely acknowledged as one of the most prominent
and powerful theories to explain how firms achieve and sustain competitive
advantage as a result of the resources they own or have under their control (Barney
2001). According to the underlying philosophy of RBT, an organization is perceived
as a bundle of valuable tangible and intangible resources, which can be combined to
generate competitive advantage (Peteraf 1993). The original RBT defines resources
as rare, inimitable, and nonsubstitutable firm-specific assets that enable a firm to
implement a value-creating strategy to generate rents (Barney 1991). This concept
was later split to distinguish between resource-picking and capability-building, two
distinct facets that are central to RBT. Resource-picking encompasses activities of
identifying and purchasing or controlling resources that are perceived as being of
strategic value, while capability-building is concerned with the orchestration and
management of these resources into strategically useful assets (Makadok 2001).
Amit and Schoemaker (1993) define resources as tradable and nonspecific firm
assets, and capabilities as nontradable firm-specific abilities to integrate, deploy, and
utilize other resources within the firm. Makadok (2001) further elaborates on the
distinction between resource-picking and capability-building in his seminal work.
According to the author, resource-picking is an important aspect since it not only
helps the firm acquire good resources, but is also important for the economic impact
of the firm by avoiding potentially poor or unworthy resources. Capability-building,
on the other hand, is concerned with activities that relate to deploying these
resources in combination with other organizational processes for the creation of
intermediate goods, which can potentially provide enhanced productivity and
strategic flexibility. Thus, resources represent the input of the production process
while a capability is the capacity to deploy these resources in the most strategically
558 P. Mikalef et al.
123
fit way. A firm’s resources and capabilities are commonly referred to as assets (Amit
and Schoemaker 1993).
Resources and capabilities are the core components of RBT, and have received a
great deal of attention in past empirical studies (Akter et al. 2016b). A characteristic
of resources is that they cannot generate any business value by themselves, but
require action to be leveraged strategically. This is indicated by Grant’s (1991)
description of resources as nouns, because they can lie dormant like an idle plant or
unused knowledge until they are needed, and can be identified independently of
their use (Wu et al. 2010). Hence, a resource is something that a firm has access to,
rather than something it can do (Größler and Grübner 2006). Several types of
resources have been suggested in the extant literature; nevertheless, one of the most
adhered-to classifications is that of Grant (1991). According to this categorization,
resources can be divided into tangible (e.g. financial and physical resources), human
skills (e.g. employees’ knowledge and skills), and intangible (e.g. organizational
culture and organizational learning) types (Grant 1991). This classification has been
predominantly followed in the IS capability literature (Bharadwaj 2000; Aral and
Weill 2007; Ravichandran and Lertwongsatien 2005).
Capabilities are described as high-level routines (or a collection of routines), with
routines consisting of learned behaviors that are highly patterned, repetitious or quasi-
repetitious, and founded in part in tacit knowledge (Winter 2003). Organizational
capabilities can be purposely built by focusing on the complex interactions between a
firm’s resources and competencies, and are therefore more complex and difficult to
imitate than just core resources (Grant 1996). According to Teece et al. (1997),
capabilities cannot easily be bought; they must be built. A basic premise of RBT is that
the capability-building process can only take place following acquisition of a resource;
therefore, developing capabilities is dependent on, and confined under, the types of
resources a firm decides to accumulate. The conversion of resources into potentially
strategic assets via the development of firm-specific capabilities has been the subject
of considerable scholarly attention (Sirmon et al. 2011). The resource orchestration
perspective attempts to explain the role of managers in terms of how resources are
transformed into capabilities, and what necessary actions are required to effectively
structure, bundle, and leverage them. This process-oriented view, which emphasizes
the conversion of resources into capabilities, is seldom addressed and is largely
affected by the heterogeneity of firms’ contexts (Barney et al. 2011).
In terms of the form that capabilities can take, previous research in the area of
strategic management has made great strides in developing and refining the different
types of capabilities. It is generally agreed that capabilities operate quite differently
from one another, and result in varying levels of competitive advantage and firm
performance based on a number of internal and external factors (Hoopes and
Madsen 2008). Grounded in the idea that firms must be both stable enough to
continue to deliver value in their own distinctive way, and agile and adaptive
enough to restructure their value proposition when circumstances demand it, there is
a well-documented distinction between operational (ordinary) and dynamic
capabilities (Drnevich and Kriauciunas 2011). Nevertheless, the resources owned
or controlled by the firm are imperative in determining what types of capabilities a
firm can develop, and of what value they will be (Wu 2007).
Big data analytics capabilities: a systematic literature… 559
123
RBT has been extensively applied to the IT context under the notion of IT
capabilities (Bharadwaj 2000). IT literature recognizes that competence in
leveraging IT-based resources in combination with other organizational resources
is a source of competitive and advantage (Pavlou and El Sawy 2006). Past empirical
studies have employed the notion of IT capabilities to demonstrate its direct (Bhatt
and Grover 2005) or indirect impact on performance outcomes (Wang et al. 2012).
The main premise adopted in these studies is that in order to develop a robust IT
capability, it is necessary for a firm to have invested in all the necessary resources
(Wade and Hulland 2004). In the context of big data, it is important to identify the
different types of resources, since the level of their infusion in various business
functions can be a source of competitive differentiation (Davenport 2006). A
conceptual framework of RBT posits that in order for a resource or capability to be a
source of competitive advantage, it must fulfill the criteria of value, rarity,
inimitability, and nonsubstitutability (i.e. so-called VRIN attributes) (Barney 1991).
When these resources and their related activity systems have complementarities,
they are more prone to lead to competitive advantage (Eisenhardt and Martin 2000).
4.2 The dynamic capabilities view of the firm
Over the past decade, the DCV of the firm has emerged as one of the most
influential theoretical perspectives in the study of strategic management (Schilke
2014). Extending the resource-based view of the firm, which posits that a firm may
achieve sustained competitive advantage based on the bundles of resources and
capabilities it has under its control, DCV attempts to explain how a firm maintains a
competitive advantage in changing environments (Priem and Butler 2001). This
shift has been ignited by commentaries from many researchers that RBT does not
adequately explain why certain firms attain a competitive advantage in situations of
rapid and unpredictable change where resources and capabilities are subject to
erosion (Eisenhardt and Martin 2000). Originating from the Schumpeterian logic of
creative destruction, dynamic capabilities enable firms to integrate, build, and
reconfigure their resources and capabilities in the face of changing conditions
(Teece et al. 1997). In essence, dynamic capabilities reformulate the way a firm
operates and competes in the market—a process referred to as evolutionary fitness
(Helfat and Peteraf 2009). Several alternative conceptualizations of dynamic
capabilities have subsequently been presented. Some follow an approach closer to
the resource-based view, which stresses the importance of strategic management
(Teece and Pisano 1994), while others approximate the logic of evolutionary
economics, which enunciates the role of routines, path dependencies, and
organizational learning (Barreto 2010).
Despite considerable variation in defining dynamic capabilities, a growing
consensus in the literature describes them as a set of identifiable and specific
routines that have often been the subject of extensive empirical research in their
own right (Eisenhardt and Martin 2000). This approach seems to be gaining
momentum in empirical studies, since it is feasible to identify and prescribe a set of
operating routines that jointly constitute firm-level dynamic capabilities (Zollo and
Winter 2002; Pavlou and El Sawy 2011). These routines are commonly recognized
560 P. Mikalef et al.
123
as learned, highly patterned, and repetitious, directed towards independent corporate
actions (Winter 2003). Consequently, to better understand dynamic capabilities it is
feasible to emphasize the set of routines that underpin them, commonly referred to
as capabilities. In the context of IS literature, several studies have examined how IT
infused in organizational capabilities can help firms renew or reconfigure their
existing mode of operating (Pavlou and El Sawy 2006; Wang et al. 2012; Mikalef
et al. 2016a, b; Mikalef and Pateli 2017). This perspective follows the logic
proposed by Henderson and Venkatraman (1993), who stressed that alignment as a
dynamic capability is not an ad-hoc event, but rather a process of continuous
adaption and change. As such, they argued that ‘no single IT application—however
sophisticated and state of the art it may be could deliver a sustained competitive
advantage.’ Rather, what is important is to infuse IT investments into the
organizational fabric (Kohli and Grover 2008; Kim et al. 2011) (Table 5).
4.3 Resources of a big data analytics capability
While the published research on BDA capability is limited, some studies have
focused on the resources necessary to develop such capability. Although resources
are of very limited value without the underlying ability to orchestrate and leverage
them, they are fundamental building blocks in the formation of a firm’s overall BDA
capability. It is therefore important to recognize the core resources and examine the
most important debate concerning each of these as described by empirical research.
By doing so, it is possible to provide a synthesis of findings that can guide practical
support in big data deployments, and also identify underexplored areas of research
that warrant further examination. The majority of studies to date have discussed the
resources and processes that need to be used to leverage big data strategically, but
have not offered much insight into how firms can develop a strong BDA capability
(Gupta and George 2016). Building on the foundations of RBT, and work in the
field of IT management that employs the theory, we present the main resources that
allow firms to develop a BDA capability. These are divided into three main
categories: tangible resources (e.g. infrastructure, IS, and data), intangible resources
(e.g. data-driven culture, governance, social IT/business alignment), and human
skills and knowledge (e.g. data analytics knowledge, and managerial skills).
4.3.1 Tangible resources
In the context of developing a BDA capability, perhaps the core resource is the data
itself. As previously mentioned, the defining characteristics of big data is their
volume, variety, and velocity (Chen and Zhang 2014). However, it is frequently
mentioned that IT strategists and data analysts are particularly concerned with the
quality of the data they analyze (Brinkhues et al. 2014). Although traditionally firms
analyzed enterprise-specific structured data, the diversity and breadth of data
sources that contemporary firms leverage render the aspect of quality highly
important (Ren et al. 2016). Data quality is regarded as a critical resource, and is
defined in terms of completeness, accuracy, format, timeliness, reliability, and
perceived value (Brinkhues et al. 2014; Ren et al. 2016). In a heavily data-oriented
Big data analytics capabilities: a systematic literature… 561
123
economy, data resources that present the previously mentioned characteristics have
been argued to be necessary for a firm to build competitive advantage (Kiron et al.
2014). Wamba et al. (2015) stress the importance of having availability and
integrating data from various sources, which traditionally may be siloed due to
existing IT architectures. The issue of availability of data is also mentioned by
Mikalef et al. (2017), who find that it is common for companies to purchase data in
order to complement their analytics and gain more insights into their customers and
Table 5 Key definitions
Concept Definition Author(s) and date
Asset Anything tangible or intangible the firm can use in its
processes for creating, producing, and/or offering its
products (goods or services) to a market
Wade and Hulland
(2004)
Resource Stocks of available factors that the organization owns or
controls
Amit and Schoemaker
(1993)
Capability A firm’s capacity to deploy resources, usually in
combination, using organizational processes, to effect a
desired end. They are information-based, tangible, or
intangible processes that are firm-specific and are
developed over time through complex interactions
among the firm’s resources
Amit and Schoemaker
(1993)
Operational
capability
Generally involves performing an activity, such as
manufacturing a particular product, using a collection of
routines to execute and coordinate the variety of tasks
required to perform the activity
Helfat and Peteraf
(2003)
Dynamic
capability
Can be disaggregated into the capacity (a) to sense and
shape opportunities and threats, (b) to seize
opportunities, and (c) to maintain competitiveness
through enhancing, combining, protecting, and, when
necessary, reconfiguring the business enterprise’s
intangible and tangible assets
Teece (2007)
Competitive
advantage
An enterprise has a competitive advantage if it is able to
create more economic value than the marginal
(breakeven) competitor in its product market
Peteraf and Barney
(2003)
Sustained
competitive
advantage
When a firm has a competitive advantage and other firms
are unable to duplicate the benefits of this strategy
Barney (1991)
VRIN VRIN resources are valuable, rare, inimitable, and
nonsubstitutable. VRIN-ness implies that resources are
heterogeneously distributed among firms. Valuable, rare
resources may be sources of competitive advantage, but
unless they are also inimitable and nonsubstitutable, that
competitive advantage will not be sustained
Barney (1991), Peteraf
and Barney (2003)
IT resources Commodity-like assets that are widely available and can
be purchased from the factor market
Cragg et al. (2011)
IT capability The ability to mobilize and deploy IT-based resources in
combination, or copresent, with other resources and
capabilities
Bharadwaj (2000)
562 P. Mikalef et al.
123
operations. A similar phenomenon is also noted in a recent report of MIT Sloan
Management Review (Ransbotham and Kiron 2017), in which it is highlighted that
firms that share data and form alliances based on such resources tend to be more
innovative. This aspect signifies the importance that the variety and diversity of data
sources have in order to derive any meaningful insights and direct strategic
initiatives.
While data itself is a core resource, it is also important for firms to possess an
infrastructure capable of storing, sharing, and analyzing data. Big data call for novel
technologies that are able to handle large amounts of diverse and fast-moving data
(Gupta and George 2016). One of the main characteristics of such data is that it is in
an unstructured format and requires sophisticated infrastructure investments in order
to derive meaningful and valuable information (Ren et al. 2016). Some scholars
examine the big data infrastructure of firms in terms of the investments made in
specific technologies (Kamioka and Tapanainen 2014), while others focus on
features of the technology itself (Akter et al. 2016a; Wamba et al. 2015; Gupta and
George 2016; Garmaki et al. 2016). In particular, scalability and connectivity are
cited as important, since the data accumulated and processes used fluctuate
continuously. Nevertheless, it is noted by many executives that infrastructure is not
a major issue for most firms, since the technology itself has extended beyond the
requirements of analytics (Mikalef et al. 2017).
With big data, novel software and IS have emerged that facilitate distributed
storage on nonrelational databases (e.g. Hadoop, Apache Cassandra, MongoDB,
Monet, and Hazelcat), parallel processing of massive unstructured datasets, and
visualization and decision aiding (Gupta and George 2016). These technologies
extend traditional ones that were built to process data in batches, enabling the
processing of continuous flows of information in real time (Wamba et al. 2015).
Despite these differences, the systems and software employed to analyze big data
follow the same principle as that of business intelligence (Lim et al. 2013). In
essence, the value of big data analysis and visualization tools is that they transform
raw data and provide business managers and analysts with appropriate information
to improve decision making (Wixom et al. 2011). Currently, there are multiple
software tools capable of facilitating requirements from very diverse data sources,
so trying to predict which ones will prevail is risky and relatively insubstantial
(Vidgen et al. 2017) (Table 6).
4.3.2 Intangible resources
Keeping up to date in terms of knowledge and skills, and effectively coordinating
activities, resources, and tasks, is highly dependent on the capacity to forge
networks internally and externally of the firm (Ravichandran and Lertwongsatien
2005). Intangible resources therefore reflect ties, structures, and roles established to
manage the different types of resources. Governance is one of the most frequent
terms used to encapsulate all the activities and decision-appropriation mechanisms
related to IT resources (Sambamurthy and Zmud 1999). Recognizing the growing
importance of managing large volumes of information, Tallon et al. (2013) proposed
a framework specifically for uncovering the structures and practices used to govern
Big data analytics capabilities: a systematic literature… 563
123
information artifacts. Their framework distinguishes governance practices into three
types: structural (assigning responsibilities, directing, and planning), procedural
(shaping user behaviors through value analysis, cost control, and resource
allocation), and relational (business–IT partnerships, idea exchange, and conflict
resolution). Espinosa and Armour (2016) define BDA governance as the approach
that analytic-based organizations use to define, prioritize, and track analytic
Table 6 Tangible resources
Resource type Characteristics Authors and date
Tangible
Data Accuracy
Timeliness
Reliability
Security
Confidentiality
Completeness
Currency
Volume
Variety
Velocity
Integration
Brinkhues et al. (2014), Chae et al. (2014), Erevelles et al.
(2016), Gupta and George (2016), Kamioka and Tapanainen
(2014), Olszak (2014), Ren et al. (2016), Wamba et al.
(2015), Phillips-Wren et al. (2015) and Vidgen et al. (2017)
Infrastructure Connectivity
Compatibility
Modularity
Agility
Large-scale,
unstructured
databases
Cloud services
Reliability
Adaptability
Integration
Accessibility
Response
Brinkhues et al. (2014), Akter et al. (2016a), Erevelles et al.
(2016), Garmaki et al. (2016), Gupta and George (2016),
Kamioka and Tapanainen (2014), Olszak (2014) and Ren
et al. (2016)
Software
and IS
Integrated analytics
systems
Security and risk-
management service
Data-management
service
Open software
Reporting and
visualization
systems
Bekmamedova and Shanks (2014), Erevelles et al. (2016),
Garmaki et al. (2016), Gupta and George (2016) and Olszak
(2014)
564 P. Mikalef et al.
123
initiatives, as well as to manage different types and categories of data related to
analytics. As such, BDA governance represents the rules and controls that
participants must comply with when performing relative tasks. The emphasis on big
data and information governance is largely attributed to the strategic importance
that it holds in contemporary enterprises. Similarly, numerous researchers note the
importance of establishing governance schemes for big data (Cao and Duan 2014b;
Garmaki et al. 2016), while others recognize it as one of the main reasons why firms
fail to leverage their data effectively (Posavec and Krajnovic 2016). A recurring
finding that concerns the effectiveness of governance, however, is that it must
follow a top-down approach, requiring commitment to data-driven decisions from
top management (Vidgen et al. 2017).
An additional intangible resource that is particularly important in driving the
adoption of big data and the development of firm-wide BDA capability is a data-
driven culture (Cao and Duan 2014a). The notion of a data-driven culture is adopted
from organizational culture, which is a highly complex concept to understand and
describe (Gupta and George 2016). In firms engaging in big data projects, a data-
driven culture has been noted as an important factor in determining their overall
success and continuation (LaValle et al. 2011). The main argument for the
importance of a data-driven culture is that although many companies implement big
data projects, the vast majority rely not on the information extracted from data
analysis, but rather on managerial experience or intuition (Provost and Fawcett
2013). This requires that organizational members, including top-level executives,
middle-level managers, and even lower-level employees, make decisions based on
information extracted from data (Gupta and George 2016). Aspects that contribute
towards a data-driven culture include prioritizing BDA investments, top manage-
ment support in formulating decisions based on BDA, and a fact-based operating
culture (Lamba and Dubey 2015; Olszak 2014; Kamioka and Tapanainen 2014). It
is then critical that the importance of data-driven decision making is imprinted in
the organization through specific practices (Mikalef et al. 2017). In fact, it is
frequently cited that organizations that are successful with BDA are those that have
managed to instill the importance of data-driven insights to a breadth of
departments. This alleviates siloed units and enables a greater depth and richness
of data to be analyzed, while also allowing for dispersed organizational units to
work collaboratively towards analytics-generated insights (Mikalef et al. 2017)
(Table 7).
4.3.3 Human skills and knowledge
The capacity to utilize big data technologies and tools such as those mentioned
above, and to make strategic decisions based on outcomes, is highly dependent on
the skills and knowledge of the human resources. These can be further divided into
technical knowledge (e.g. database management, data retrieval, programming
knowledge such as MapReduce, and cloud service management), business
knowledge (e.g. decision making heavily routed within the firm, strategic foresight
for big data deployments, and application of insights extracted), relational
knowledge (e.g. communication and collaboration skills between employees of
Big data analytics capabilities: a systematic literature… 565
123
different backgrounds), and business analytics knowledge (e.g. mathematical
modeling, simulation and scenario development, and interactive data visualization).
In a highly influential article, Davenport and Patil (2012) address the important role
that the emerging job of the data scientist will have in the context of big data. While
one of the most critical aspects of data science is the ability to think analytically
about data, such skill is not only important for the data scientist, but for employees
throughout the organization (Prescott 2014). In effect, the data scientist is capable of
understanding business problems and utilizing relevant data sources to generate
insights based on models and visualization tools.
Recognizing the importance of the data scientist in contemporary firms, some
studies have even proposed methods to redesign IS curriculums (Jacobi et al. 2014).
This lack of personnel with the appropriate skills is also noted in numerous studies,
and constitutes a major constraint in realizing the full potential of these technologies
(Tambe 2014). A report by McKinsey Global Institute concludes that by 2018 there
will be a shortage of talent necessary for organizations to take advantage of big data,
with an estimate of 140,000–190,000 positions for which no trained personnel will
be available (Domingue et al. 2014). While it is still vague and unclear what the
critical skills that a data scientist must have are, some definitions help to clarify this.
According to Mohanty et al. (2013), data scientists are practitioners of the analytics
models solving business problems. They incorporate advanced analytical
approaches using sophisticated analytics and data visualization tools to discover
patterns in data. The core attributes of the data scientist have been distilled to having
Table 7 Intangible resources
Resource
type
Characteristics Authors and date
Intangible
Governance Control
Coordination and
monitoring
Business–IT alignment
Decision-rights
appropriation
Big data solution
assessment and
validation
Business vision and
planning
Policy and rule structures
Olszak (2014), Garmaki et al. (2016), Akter et al. (2016a),
Cao and Duan (2014b), Erevelles et al. (2016), Tallon et al.
(2013), Espinosa and Armour (2016), Phillips-Wren et al.
(2015), Mikalef et al. (2017), and Vidgen et al. (2017)
Data-driven
culture
Prioritizing BA
investments
Top management support
Fact-based and learning
culture
Davenport et al. (2001), Davenport (2013), Kiron et al.
(2014), Kiron and Shockley (2011), Gupta and George
(2016), Lamba and Dubey (2015), Olszak (2014),
Kamioka and Tapanainen (2014) and Mikalef et al. (2017)
566 P. Mikalef et al.
123
entrepreneurial and business domain knowledge, computer science skills, effective
communication skills, ability to create valuable and actionable insights, inquisi-
tiveness and curiosity, and knowledge of statistics and modeling (Chatfield et al.
2014).
Nevertheless, while the center of attention has been placed on the data scientist
primarily due to the novelty of the role, other skills and knowledge sets are
necessary in employees of firms engaging in BDA. Of particular relevance are
technical skills such as those of the big data engineers, who are able to acquire store,
cleanse, and code data from multiple sources and of various formats (Mikalef et al.
2017). Similarly, big data architects accommodate such technical knowledge by
being responsible for developing blueprint plans of the data sources, as well as the
appropriate technologies to leverage their potential. Due to the fusion of business
and IT departments in BDA firms, the importance of a liaison person has emerged;
that is, a person capable of bridging the siloed departments and making them work
collaboratively (Akter et al. 2016a). The necessary skills for such employees include
a good understanding of what each department/unit is doing, as well as an ability to
communicate with each and build fused teams (Mikalef et al. 2017). Finally, having
a good understanding of the goals and directions of the firm, as well as knowing
how to measure and improve critical key performance indicators (KPIs), is of
paramount importance since, in most cases, BDA are grounded on an existing
problem. Therefore, an ability to identify this problem and improve by means of big
data-generated insight is an important aspect of the knowledge that business
executives and data analysts should have (Gupta and George 2016) (Table 8).
4.4 Areas of big data analytics
Apart from the core resources that are required to develop a BDA capability, several
studies have examined the areas in which big data initiatives can be leveraged, as
well as overall firm performance gains (Akter and Wamba 2016). The main premise
is that although a BDA capability comprises mostly similar aspects that need to be
taken into account independently of context, the way in which this capability is
applied has considerable diversity. For instance, several studies note that companies
that belong in the media and news industry apply their BDA capabilities towards
personalizing content toward their customers and delivering tailored-made news and
suggestions, while in the oil and gas industry there are several applications geared
towards risk assessment and maintenance (Mikalef et al. 2017). Furthermore,
considerable heterogeneity of BDA applications has been observed within
industries, which can potentially lead to differentiated business value outcomes
(Akter et al. 2016a). To this end, several recent studies attempt to examine the
influence of BDA capabilities on enabling various forms of organizational
capabilities (Xu et al. 2016; Pappas et al. 2016; Wamba et al. 2017).
These studies show that a BDA capability can be directed towards strengthening
both operational (Chae et al. 2014) and dynamic (Erevelles et al. 2016) capabilities
of a firm. Wamba et al. (2015) demonstrate that the types of value creation for big
data initiatives can be divided into creating transparency (Meredith et al. 2012;
Bärenfänger et al. 2014), enabling experimentation to discover needs and improve
Big data analytics capabilities: a systematic literature… 567
123
T a b le
8 H u m a n re so u rc e s
R e so u rc e ty p e
C h a ra c te ri st ic s
A u th o rs
a n d d a te
H u m a n sk il ls a n d k n o w le d g e
T e c h n ic a l k n o w le d g e
P ro g ra m m in g
T e c h n ic a l in fr a st ru c tu re
m a n a g e m e n t
M a p R e d u c e
U n st ru c tu re d d a ta
m a n a g e m e n t
D a ta
c o ll e c ti o n /i n te g ra ti o n
A k te r e t a l. (2 0 1 6 a ), E lb a sh ir e t a l. (2 0 1 3 ),
G a rm
a k i e t a l. (2 0 1 6 ), G u p ta
a n d G e o rg e
(2 0 1 6 ), K a m io k a a n d T a p a n a in e n (2 0 1 4 ),
O ls z a k , (2 0 1 4 ), a n d M ik a le f e t a l. (2 0 1 7 )
B u si n e ss
k n o w le d g e
B u si n e ss
st ra te g y
K P Is
B u si n e ss
p ro c e ss e s
C h a n g e m a n a g e m e n t
A k te r e t a l. (2 0 1 6 a ), E re v e ll e s e t a l. (2 0 1 6 ),
E lb a sh ir e t a l. (2 0 1 3 ), G u p ta
a n d G e o rg e
(2 0 1 6 ), O ls z a k (2 0 1 4 ), G a rm
a k i e t a l.
(2 0 1 6 ), a n d O ls z a k (2 0 1 4 )
R e la ti o n a l k n o w le d g e
C o m m u n ic a ti o n sk il ls
T e a m
b u il d in g
A k te r e t a l. (2 0 1 6 a ), G a rm
a k i e t a l. (2 0 1 6 )
a n d M ik a le f e t a l. (2 0 1 7 )
B u si n e ss
a n a ly ti c s
S ta ti st ic a l a n a ly si s
F o re c a st in g
Q u e ry
a n d a n a ly si s
P re d ic ti v e m o d e li n g
O p ti m iz a ti o n
M o d e l m a n a g e m e n t
S im
u la ti o n a n d sc e n a ri o d e v e lo p m e n t
B u si n e ss
re p o rt in g /K P Is /d a sh b o a rd s
W e b a n a ly ti c s
S o c ia l m e d ia
a n a ly ti c s
In te ra c ti v e d a ta
v is u a li z a ti o n
T e x t, a u d io , v id e o a n a ly ti c s
D a ta
a n d te x t m in in g
D a v e n p o rt e t a l. (2 0 0 1 ), L a V a ll e e t a l.
(2 0 1 1 ), C h e n e t a l. (2 0 1 6 ), C a o a n d D u a n
(2 0 1 4 b ), C h a e e t a l. (2 0 1 4 ), E re v e ll e s e t a l.
(2 0 1 6 ), G a lb ra it h (2 0 1 4 ), G a rm
a k i e t a l.
(2 0 1 6 ), L a m b a a n d D u b e y (2 0 1 5 ), O ls z a k
(2 0 1 4 ) a n d V id g e n e t a l. (2 0 1 7 )
568 P. Mikalef et al.
123
performance (Brinkhues et al. 2014; Bärenfänger et al. 2014), segmenting
populations (Kowalczyk and Buxmann 2014), enhancing or replacing human
decision making (Meredith et al. 2012; Cao et al. 2015; Brinkhues et al. 2014;
Bärenfänger et al. 2014), and innovating new business models, products, and
services (Jelinek and Bergey 2013). Hence, it is important when considering the
potential of BDA capabilities to take into account their area of application. Utilizing
BDA to improve process efficiency is most likely to have significantly less impact
on a firm’s competitive position compared to utilizing them to detect new customer
segments or come up with new business models. Nevertheless, even if BDA
capabilities are leveraged in core strategic areas, their value is likely to be
dependent and contingent upon multiple factors, which will be further elaborated in
the subsequent section.
5 Discussion
Despite the hype that surrounds big data, the business potential and mechanisms
through which it results in competitive performance gains have remained
largely underexplored to date in empirical studies. By conducting a systematic
literature review and documenting what is known to date, it is possible to
identify prominent themes of research that are of high relevance. We do so by
defining six thematic areas of research, as depicted in the research framework in
Fig. 2, and provide some suggestions on how scholars could approach these
problems.
5.1 Theme 1: resource orchestration of big data analytics
While considerable effort has been made to define the building blocks of a firm’s
BDA capability, little is known so far about the processes and structures
necessary to orchestrate these resources into a firm-wide capability. In other
words, literature has been very detailed on the resource-picking aspect of BDA,
but less so on the activities that need to be put into place to develop the
capability. Prior literature on IT–business value has shown that competence in
orchestrating and managing such resources is a prerequisite to developing the
capacity so as to leverage these resources strategically (Cragg et al. 2011; Wang
et al. 2012). According to the framework of resource orchestration, structuring
resources towards the building of a capability consists of acquiring, accumu-
lating, and divesting resources (Sirmon et al. 2011). Therefore, it is important
for researchers to examine the capability-building process, since it is likely that
firms with similar resources will exert highly varied levels of BDA capabilities.
Similarly, firms with same levels of BDA capabilities may develop them in
dissimilar ways, since their value may be contingent upon several internal and
external factors (Mikalef et al. 2015).
Big data analytics capabilities: a systematic literature… 569
123
5.2 Theme 2: decoupling big data analytics capability from big data- enabled capabilities
Although it is clear that a BDA capability refers to a firm’s proficiency in
orchestrating and managing its big data-related resources, it is important to
differentiate between the firm’s capacity to utilize its BDA capability towards
insight generation of organizational-level capabilities. As such, a firm can have
developed a strong BDA capability but only utilize it towards a specific type of
operational capability (e.g. marketing). Therefore, the assumption that a BDA
capability will enhance several organizational capabilities simultaneously is
misleading. It is highly likely that industry and other contextual factors influence
firms’ decisions to leverage their BDA capabilities in order to gain insight in
relation to different organizational capabilities. Again, the means by which they
choose to leverage their BDA capabilities could differ significantly and could
possibly result in variation in terms of performance gains. Consequently, it is critical
to gain a deeper understanding of the specifics of each capability, since the high-
level abstractions noted in strategic management literature may conceal the reality
of how the capability is leveraged by means of BDA (Mikalef and Pateli 2016).
5.3 Theme 3: bounded rationality of big data analytics
One of the assumptions of BDA, which is not discussed very frequently, has to do
with the limitations of insights that can be derived from big data itself. While BDA
may enable a more data-driven decision-making approach, the types of insights that
can be derived are bounded by the amount, variety, and quality of available data.
Consequently, a frequently observed phenomenon is that firms try to access a
diverse set of data from open sources. In some cases, it is even noted that firms are
forming strategic alliances in which the exchanged resources are datasets and
customer information (Ransbotham and Kiron 2017). Hence, one of the conditions
that should be taken into account when examining the strategic potential of big data
Fig. 2 Big data analytics research framework
570 P. Mikalef et al.
123
is the availability of data. In a similar vein, the different forms of collaborative
agreements with regards to data exchange and their resulting business value are
posited to be an area of increased interest. It is highly probable that the boundaries
of the insight that firms can develop, and, subsequently, the types of competitive
actions that they launch, are restricted by the availability of data.
5.4 Theme 4: turning big data analytics insights into action
To realize the value of BDA, it is necessary not only to put them into action in the
generation of data-driven information for specific organizational capabilities, but
also to take action to harness the insights. While some studies assume that
leveraging BDA capabilities is sufficient to provide business value, it is important to
examine the mechanisms of inertia that act in inhibiting their value. In a recent
literature review, Besson and Rowe (2012) indicate that when it comes to
organizational transformation there are five main types of inertia that hinder the
value of IS. These include negative psychology inertia, socio-cognitive inertia,
socio-technical inertia, economic inertia, and political inertia. The aspects of inertia
can work at multiple stages within the development of a BDA capability, and also
after they have resulted in insights. Nevertheless, turning BDA into action, and
subsequently business value, may also be dependent upon the external environment.
In highly dynamic and turbulent environments, the value of insight may be
diminished by scarce resources or competitors launching competitive actions in
short cycles. Hence, it is important to examine the confluence of the competitive
environment when considering the business value of BDA-derived insight.
5.5 Theme 5: trust of top managers in big data analytics insights
One specific form of distrust in the value and accuracy of BDA can be detected on
the top management level. While managers may be positive about investing in BDA
capabilities, when it comes to decision making they may feel that their intuition is
more accurate than the analysis performed on big datasets. This phenomenon has
been studied under the prism of dual process theory in the organizational context
(Hodgkinson and Healey 2011). The main premise is that managers’ emotional and
cognitive responses may override the insights of BDA, thereby reducing their
potential value. This dichotomy between reflexive systems inherent in top
managers, such as implicit stereotyping and automatic categorization, and reflective
systems, such as those present in BDA that allow logical reasoning and planning,
are argued to be important when measuring the impact of such investments.
5.6 Theme 6: business value measurement
The issue of determining the right measurement indices by which to assess the value
of IT is prevalent throughout IS research (Schryen 2013). Similarly, the hype
surrounding BDA may cause managers and academics to overestimate the potential
business value of BDA, or develop biased performance metrics to benchmark BDA
effectiveness. It is therefore important to construct specific performance measures
Big data analytics capabilities: a systematic literature… 571
123
depending on a number of contextual factors, as well as on the area in which BDA
are deployed. Furthermore, it is critical to identify metrics that take into account the
competitive landscape, since in highly uncertain and dynamic markets the value of
BDA may be reduced due to competitors following similar strategies or scarce
resources inhibiting response formation. As such, we highly encourage researchers
to examine a multitude of not only objective, but also subjective, value measures.
6 Conclusion
An ever-increasing number of companies are attempting to use big data and
business analytics in order to analyze available data and aid decision making. For
these companies, it is important to leverage the full potential that big data and
business analytics can offer with the aim of gaining competitive advantage.
Nevertheless, since big data and business analytics are a relatively new technolog-
ical and business paradigm, there is little research on how to effectively manage
them and leverage them. While early studies have shown the benefits of using big
data in different contexts, there is a lack of theoretically driven research on how to
utilize these solutions in order to gain a competitive advantage. This work identifies
the need to examine BDA through a holistic lens. We thereby focus on summarizing
what we already know and pinpoint themes on which we still have limited empirical
understanding.
To this end, this study proposes a research framework that is based on prior
literature in the area of IT–business value, as well as on concepts from strategic
management and management IS literature. The framework provides a reference for
the broader implementation of big data in the business context. While the elements
present in the research framework are on a high level and can be interpreted as quite
abstract, they are purposefully described in such a manner that they can be adapted
depending on the company at hand. This poses a novel perspective on big data
literature, since the vast majority focuses on tools, technical methods (e.g. data
mining, textual analysis, and sentiment analysis), network analytics, and infras-
tructure. Hence, the proposed framework contributes to big data and business
strategy literature by covering the aforementioned gap. It is more important for
managers and decision makers to learn how to implement big data and business
analytics in their competitive strategies than to simply perform raw data analysis on
large datasets without a clear direction of where this contributes to the overall
business strategy.
Furthermore, this study argues that the main source of competitive edge,
especially in highly dynamic and turbulent environments, will stem from companies
being able to reinforce their organizational capabilities through targeted use of big
data and business analytics. This of course does not lessen the importance of big
data resources and capabilities, since their availability and VRIN characteristics can
determine the strength of the associated insights that augment organizational
capabilities (Bowman and Ambrosini 2003; Meyer-Waarden 2016). The concepts
used in the proposed framework may help managers to better understand, plan, and
organize the process of implementing BDA within a business strategy. In addition,
572 P. Mikalef et al.
123
the framework can be used as a roadmap for gradually maturing the BDA capability
of a firm and deriving increased value from such initiatives.
This paper offers a theoretical framework on how to increase business value and
competitive performance through targeted application of big data. Future studies
should empirically test and evaluate this framework using surveys, interviews,
observation, focus groups with experts (e.g. managers, decision makers) and with
customers, as well as case studies from the industry. In addition, both qualitative
and quantitative methods of data collection should be employed. For each different
type of data, more than one method of analysis should be used (e.g. structural
equation modeling, qualitative comparative analysis). The main argument of this
systematic literature review is that the value of big data does not solely rely on the
technologies used to enable them, but is apparent through a large nexus of
associations that are eventually infused with organizational capabilities. Strength-
ening these capabilities by virtue of big data is what will lead to competitive
performance gains, and is contingent upon multiple internal and external factors.
References
Abbasi A, Sarker S, Chiang RH (2016) Big data research in information systems: toward an inclusive
research agenda. J Assoc Inf Syst 17(2):1–32
Agarwal R, Dhar V (2014) Editorial-big data, data science, and analytics: the opportunity and challenge
for IS research. Inf Syst Res 25(3):443–448
Akter S, Wamba SF (2016) Big data analytics in E-commerce: a systematic review and agenda for future
research. Electron Mark 26(2):173–194
Akter S, Wamba SF, Gunasekaran A, Dubey R, Childe SJ (2016a) How to improve firm performance
using big data analytics capability and business strategy alignment? Int J Prod Econ 182:113–131
Akter S, Wamba SF, Gunasekaran A, Dubey R, Childe SJ (2016b) How to improve firm performance
using big data analytics capability and business strategy alignment? Int J Prod Econ 182:113–131
Amit R, Schoemaker PJ (1993) Strategic assets and organizational rent. Strateg Manag J 14(1):33–46
Aral S, Weill P (2007) IT assets, organizational capabilities, and firm performance: how resource
allocations and organizational differences explain performance variation. Organ Sci 18(5):763–780
Bärenfänger R, Otto B, Österle H (2014) Business value of in-memory technology—multiple-case study
insights. Ind Manag Data Syst 114(9):1396–1414
Barney J (1991) Firm resources and sustained competitive advantage. J Manag 17(1):99–120
Barney JB (2001) Resource-based theories of competitive advantage: a ten-year retrospective on the
resource-based view. J Manag 27(6):643–650
Barney JB, Ketchen DJ Jr, Wright M (2011) The future of resource-based theory: revitalization or
decline? J Manag 37(5):1299–1315
Barreto I (2010) Dynamic capabilities: a review of past research and an agenda for the future. J Manag
36(1):256–280
Bekmamedova N, Shanks G (2014) Social media analytics and business value: a theoretical framework
and case study. In: Proceedings of 2014 47th Hawaii international conference on system sciences
(HICSS). IEEE
Besson P, Rowe F (2012) Strategizing information systems-enabled organizational transformation: a
transdisciplinary review and new directions. J Strateg Inf Syst 21(2):103–124
Beyer MA, Laney D (2012) The importance of ‘big data’: a definition. Gartner, Stamford, pp 2014–2018
Bharadwaj AS (2000) A resource-based perspective on information technology capability and firm
performance: an empirical investigation. MISQ 24(1):169–196
Bharadwaj A, El Sawy OA, Pavlou PA, Venkatraman NV (2013) Digital business strategy: toward a next
generation of insights. MISQ 37(2):471–482
Big data analytics capabilities: a systematic literature… 573
123
Bhatt GD, Grover V (2005) Types of information technology capabilities and their role in competitive
advantage: an empirical study. J Manag Inf Syst 22(2):253–277
Bowman C, Ambrosini V (2003) How the resource-based and the dynamic capability views of the firm
inform corporate-level strategy. Br J Manag 14(4):289–303
Boyd D, Crawford K (2012) Critical questions for big data: Provocations for a cultural, technological, and
scholarly phenomenon. Inf Commun Soc 15(5):662–679
Brinkhues R, Maçada AC, Casalinho G (2014) Information management capabilities: antecedents and
consequences. In: Proceedings of Americas conference on information systems (AMCIS)
Cao G, Duan Y (2014a) A path model linking business analytics, data-driven culture, and competitive
advantage. In: European conference on information systems (ECIS)
Cao G, Duan Y (2014b) Gaining competitive advantage from analytics through the mediation of decision-
making effectiveness: an empirical study of UK manufacturing companies. In: Proceedings of the
Pacific Asia conference on information systems (PACIS), p 377
Cao G, Duan Y, Li G (2015) Linking business analytics to decision making effectiveness: a Path model
analysis. IEEE Trans Eng Manag 62(3):384–395
Chae BK, Yang C, Olson D, Sheu C (2014) The impact of advanced analytics and data accuracy on
operational performance: a contingent resource based theory (RBT) perspective. Decis Support Syst
59:119–126
Chatfield AT, Shlemoon VN, Redublado W, Rahman F (2014) Data scientists as game changers in big
data environments. In: Proceedings of the 25th Australasian conference on information systems
(ACIS)
Chen CP, Zhang CY (2014) Data-intensive applications, challenges, techniques and technologies: a
survey on Big Data. Inf Sci 275:314–347
Chen Y, Chen H, Gorkhali A, Lu Y, Ma Y, Li L (2016) Big data analytics and big data science: a survey.
J Manag Anal 3(1):1–42
Constantiou ID, Kallinikos J (2015) New games, new rules: big data and the changing context of strategy.
J Inf Technol 30(1):44–57
Cragg P, Caldeira M, Ward J (2011) Organizational information systems competences in small and
medium-sized enterprises. Inf Manag 48(8):353–363
Davenport TH (2006) Competing on analytics. Harv Bus Rev 84(1):98–107
Davenport TH (2013) Analytics 3.0. Harv Bus Rev 91(12):64–72
Davenport TH, Harris JG (2007) Competing on analytics: the new science of winning. Harv Bus Press,
Boston
Davenport TH, Patil DJ (2012) Data scientist. Harv Bus Rev 90(5):70–76
Davenport TH, Harris JG, David W, Jacobson AL (2001) Data to knowledge to results: building an
analytic capability. Calif Manag Rev 43(2):117–138
Davis CK (2014) Beyond data and analysis. Commun ACM 57(6):39–41
De Mauro A, Greco M, Grimaldi M (2015) What is big data? A consensual definition and a review of key
research topics. In: AIP Conference Proceedings, vol 1644, no 1, pp 97–104
Demchenko Y, Grosso P, De Laat C, Membrey P (2013) Addressing big data issues in scientific data
infrastructure. In: 2013 International conference on collaboration technologies and systems (CTS).
IEEE, pp 48–55
Domingue J, d’Aquin M, Simperl E, Mikroyannidis A (2014) The web of data: bridging the skills gap.
IEEE Intell Syst 1(29):70–74
Dong XL, Srivastava D (2013) Big data integration. In: Proceedings of 2013 IEEE 29th international
conference on data engineering (ICDE), pp 1245–1248)
Drnevich PL, Kriauciunas AP (2011) Clarifying the conditions and limits of the contributions of ordinary
and dynamic capabilities to relative firm performance. Strateg Manag J 32(3):254–279
Eisenhardt KM, Martin JA (2000) Dynamic capabilities: what are they? Strateg Manag J
21(10–11):1105–1121
Elbashir MZ, Collier PA, Sutton SG, Davern MJ, Leech SA (2013) Enhancing the business value of
business intelligence: the role of shared knowledge and assimilation. J Inf Syst 27(2):87–105
Erevelles S, Fukawa N, Swayne L (2016) Big data consumer analytics and the transformation of
marketing. J Bus Res 69(2):897–904
Espinosa JA, Armour F (2016) The big data analytics gold rush: a research framework for coordination
and governance. In: Proceedings of 2016 49th Hawaii international conference on system sciences
(HICSS), pp 1112–1121
Galbraith JR (2014) Organization design challenges resulting from big data. J Organ Des 3:2–13
574 P. Mikalef et al.
123
Gandomi A, Haider M (2015) Beyond the hype: Big data concepts, methods, and analytics. Int J Inf
Manag 35(2):137–144
Gantz J, Reinsel D (2012) The digital universe in 2020: big data, bigger digital shadows, and biggest
growth in the far east. IDC iView IDC Anal Future 2012:1–16
Garmaki M, Boughzala I, Wamba SF (2016) The effect of big data analytics capability on firm
performance. In: Proceedings of 20th Pacific Asia conference on information systems (PACIS)
George G, Osinga CE, Lavie D, Scott B (2016) Big data and data science methods for management
research. Acad Manag J 59(5):1493–1507
Ghasemaghaei M, Hassanein K, Turel O (2015) Impacts of big data analytics on organizations: a resource
fit perspective. In: Proceedings of 21st Americas conference on information systems (AMCIS)
Grant RM (1991) The resource-based theory of competitive advantage: implications for strategy
formulation. Calif Manag Rev 33(3):114–135
Grant RM (1996) Prospering in dynamically-competitive environments: organizational capability as
knowledge integration. Organ Sci 7(4):375–387
Größler A, Grübner A (2006) An empirical model of the relationships between manufacturing
capabilities. Int J Oper Prod Manag 26(5):458–485
Gupta M, George JF (2016) Toward the development of a big data analytics capability. Inf Manag
53(8):1049–1064
Helfat CE, Peteraf MA (2003) The dynamic resource-based view: capability lifecycles. Strateg Manag J
24(10):997–1010
Helfat CE, Peteraf MA (2009) Understanding dynamic capabilities: progress along a developmental path.
Strateg Organ 7(1):91–102
Henderson JC, Venkatraman H (1993) Strategic alignment: leveraging information technology for
transforming organizations. IBM Syst J 32(1):472–484
Higgins JP, Green S (eds) (2008) Cochrane handbook for systematic reviews of interventions, vol 5.
Wiley-Blackwell, Chichester
Hodgkinson GP, Healey MP (2011) Psychological foundations of dynamic capabilities: reflexion and
reflection in strategic management. Strateg Manag J 32(13):1500–1516
Hoopes DG, Madsen TL (2008) A capability-based view of competitive heterogeneity. Ind Corp Change
17(3):393–426
Jacobi F, Jahn S, Krawatzeck R, Dinter B, Lorenz A (2014) Towards a design model for interdisciplinary
information systems curriculum development, as exemplified by big data analytics education. In:
Proceedings of European conference on information systems (ECIS)
Jelinek M, Bergey P (2013) Innovation as the strategic driver of sustainability: big data knowledge for
profit and survival. IEEE Eng Manag Rev 41(2):14–22
Kamioka T, Tapanainen T (2014) Organizational use of big data and competitive advantage—exploration
of antecedents. In: Pacific Asia conference on information systems (PACIS), p 372
Kim G, Shin B, Kim KK, Lee HG (2011) IT capabilities, process-oriented dynamic capabilities, and firm
financial performance. J Assoc Inf Syst 12(7):487–517
Kiron D, Shockley R (2011) Creating business value with analytics. MIT Sloan Manag Rev 53(1):57–63
Kiron D, Prentice PK, Ferguson RB (2014) The analytics mandate. MIT Sloan Manag Rev 55(4):1–25
Kitchenham BA (2004) Procedures for performing systematic reviews. Joint technical report. Computer
Science Department, Keele University (TR/SE-0401) and National ICT Australia Ltd (0400011T.1)
Kitchenham BA (2007) Guidelines for performing systematic literature reviews in software engineering
version 2.3, Keele University and University of Durham, EBSE technical report
Kitchenham BA, Brereton OP, Budgen D, Turner M, Bailey J, Linkman S (2009) Systematic literature
reviews in software engineering—a systematic literature review. Inf Softw Technol 51(1):7–15
Kohli R, Grover V (2008) Business value of IT: an essay on expanding research directions to keep up
with the times. J Assoc Inf Syst 9(1):23–39
Kowalczyk DWIM, Buxmann P (2014) Big data and information processing in organizational decision
processes. Bus Inf Syst Eng 6(5):267–278
Kung L, Kung HJ, Jones-Farmer A, Wang Y (2015) Managing big data for firm performance: a
configurational approach. In: Americas conference on information systems (AMCIS)
Kwon O, Lee N, Shin B (2014) Data quality management, data usage experience and acquisition intention
of big data analytics. Int J Inf Manag 34(3):387–394
Lamba HS, Dubey SK (2015) Analysis of requirements for big data adoption to maximize IT business
value in reliability. In: 2015 4th International conference on infocom technologies and optimization
(ICRITO) (trends and future directions). IEEE, pp 1–6
Big data analytics capabilities: a systematic literature… 575
123
LaValle S, Lesser E, Shockley R, Hopkins MS, Kruschwitz N (2011) Big data, analytics and the path
from insights to value. MIT Sloan Manag Rev 52(2):21–32
Lim EP, Chen H, Chen G (2013) Business intelligence and analytics: research directions. ACM Trans
Manag Inf Syst (TMIS) 3(4):17–27
Loebbecke C, Picot A (2015) Reflections on societal and business model transformation arising from
digitization and big data analytics: a research agenda. J Strateg Inf Syst 24(3):149–157
Makadok R (2001) Toward a synthesis of the resource-based and dynamic-capability views of rent
creation. Strateg Manag J 22(5):387–401
Markus ML (2015) New games, new rules, new scoreboards: the potential consequences of big data. J Inf
Technol 30(1):58–59
McAfee A, Brynjolfsson E, Davenport TH, Patil DJ, Barton D (2012) Big data: the Manag revolution.
Harv Bus Rev 90(10):61–67
Meredith R, Remington S, O’Donnell P, Sharma N (2012) Organisational transformation through
business intelligence: theory, the vendor perspective and a research agenda. J Decis Syst
21(3):187–201
Meyer-Waarden L (2016) Big data resources, marketing capabilities, and firm performance. In:
Proceedings of the 37th international conference on information systems (ICIS)
Mikalef P, Pateli AG (2016) Developing and validating a measurement instrument of IT-enabled dynamic
capabilities. In: Proceedings of the 24th European conference on information systems (ECIS)
Mikalef P, Pateli A (2017) Information technology-enabled dynamic capabilities and their indirect effect
on competitive performance: findings from PLS-SEM and fsQCA. J Bus Res 70:1–16
Mikalef P, Pateli A, Batenburg RS, Wetering RVD (2015) Purchasing alignment under multiple
contingencies: a configuration theory approach. Ind Manag Data Syst 115(4):625–645
Mikalef P, Pateli A, van de Wetering R (2016a) IT flexibility and competitive performance: the mediating
role of IT-enabled dynamic capabilities. In: Proceedings of the 24th European conference on
information systems (ECIS)
Mikalef P, Pappas IO, Giannakos MN, Krogstie J, Lekakos G (2016b) Big data and strategy: a research
framework. In: Proceedings of the 10th mediterranean conference on information systems (MCIS)
Mikalef P, Framnes V, Danielsen F, Krogstie J, Olsen DH (2017) Big data analytics capability:
antecedents and business value. In: Proceedings of the 21st Pacific Asia conference on information
systems (PACIS)
Mohanty S, Jagadeesh M, Srivatsa H (2013) Big data imperatives: enterprise ‘‘Big Data’’ warehouse, BI
implementations and analytics. Apress, New York
Müller O, Junglas I, vom Brocke J, Debortoli S (2016) Utilizing big data analytics for information
systems research: challenges, promises and guidelines. Eur J Inf Syst 25(4):289–302
Olszak CM (2014) Towards an understanding business intelligence a dynamic capability-based
framework for business intelligence. In: 2014 Federated conference on computer science and
information systems (FedCSIS). IEEE, pp 1103–1110
Opresnik D, Taisch M (2015) The value of big data in servitization. Int J Prod Econ 165:174–184
Oracle (2012) Big data for the enterprise. Oracle, Redwood Shores
Pappas IO, Mikalef P, Giannakos MN, Krogstie J, Lekakos G (2016) Social media and analytics for
competitive performance: a conceptual research framework. In: International conference on business
information systems. Springer, Cham, pp 209–218
Pavlou PA, El Sawy OA (2006) From IT leveraging competence to competitive advantage in turbulent
environments: the case of new product development. Inf Syst Res 17(3):198–227
Pavlou PA, El Sawy OA (2011) Understanding the elusive black box of dynamic capabilities. Decis Sci
42(1):239–273
Peteraf MA (1993) The cornerstones of competitive advantage: a resource-based view. Strateg Manag J
14(3):179–191
Peteraf MA, Barney JB (2003) Unraveling the resource-based tangle. Manag Decis Econ 24(4):309–323
Phillips-Wren G, Iyer LS, Kulkarni U, Ariyachandra T (2015) Business analytics in the context of big
data: a roadmap for research. Commun Assoc Inf Syst 37(1):448–472
Posavec AB, Krajnović S (2016) Challenges in adopting big data strategies and plans in organizations. In:
Proceedings of 39th international convention on information and communication technology,
electronics, and microelectronics
Prescott EM (2014) Big data and competitive advantage at Nielsen. Manag Decis 52(3):573–601
Priem RL, Butler JE (2001) Is the resource-based ‘‘view’’ a useful perspective for strategic manag
research? Acad Manag Rev 26(1):22–40
576 P. Mikalef et al.
123
Provost F, Fawcett T (2013) Data science and its relationship to big data and data-driven decision making.
Big Data 1(1):51–59
Ransbotham S, Kiron D (2017) Analytics as a source of business innovation. MIT Sloan Management
Review. Research report. http://sloanreview.mit.edu/projects/analytics-as-a-source-of-business-
innovation/
Ravichandran T, Lertwongsatien C (2005) Effect of information systems resources and capabilities on
firm performance: a resource-based perspective. J Manag Inf Syst 21(4):237–276
Ren S, Wamba SF, Akter S, Dubey R, Childe SJ (2016) Modelling quality dynamics, business value and
firm performance in a big data analytics environment Int. J Prod Res 55(17):5011–5026
Russom P (2011) Big data analytics. TDWI Best Practices Report, Fourth Quarter 1–35
Sambamurthy V, Zmud RW (1999) Arrangements for information technology governance: a theory of
multiple contingencies. MISQ 23(2):261–290
Schilke O (2014) On the contingent value of dynamic capabilities for competitive advantage: the
nonlinear moderating effect of environmental dynamism. Strateg Manag J 35(2):179–203
Schroeck M, Shockley R, Smart J, Romero-Morales D, Tufano P (2012) Analytics: The real-world use of
big data. IBM Global Business Services 1–20
Schryen G (2013) Revisiting IS business value research: what we already know, what we still need to
know, and how we can get there. Eur J Inf Syst 22(2):139–169
Seddon JJ, Currie WL (2017) A model for unpacking big data analytics in high-frequency trading. J Bus
Res 70:300–307
Sharda R, Delen D, Turban E (2013) Business intelligence: a managerial perspective on analytics.
Prentice Hall Press, Prentice
Shuradze G, Wagner HT (2016) Towards a conceptualization of data analytics capabilities. In: 2016 49th
Hawaii international conference on system sciences (HICSS). IEEE, pp 5052–5064
Sirmon DG, Hitt MA, Ireland RD, Gilbert BA (2011) Resource orchestration to create competitive
advantage: breadth, depth, and life cycle effects. J Manag 37(5):1390–1412
Sun EW, Chen YT, Yu MT (2015) Generalized optimal wavelet decomposing algorithm for big financial
data. Int J Prod Econ 165:194–214
Tallon PP, Ramirez RV, Short JE (2013) The information artifact in IT governance: toward a theory of
information governance. J Manag Inf Syst 30(3):141–178
Tambe P (2014) Big data investment, skills, and firm value. Manag Sci 60(6):1452–1469
Teece DJ (2007) Explicating dynamic capabilities: the nature and microfoundations of (sustainable)
enterprise performance. Strateg Manag J 28(13):1319–1350
Teece D, Pisano G (1994) The dynamic capabilities of firms: an introduction. Ind Corp Change
3(3):537–556
Teece DJ, Pisano G, Shuen A (1997) Dynamic capabilities and strategic management. Strateg Manag J
18(7):509–533
Vidgen R, Shaw S, Grant DB (2017) Management challenges in creating value from business analytics.
Eur J Oper Res 261(2):626–639
Wade M, Hulland J (2004) Review: the resource-based view and information systems research: Review,
extension, and suggestions for future research. MISQ 28(1):107–142
Wamba SF, Akter S, Edwards A, Chopin G, Gnanzou D (2015) How ‘big data’ can make big impact:
findings from a systematic review and a longitudinal case study. Int J Prod Econ 165:234–246
Wamba SF, Gunasekaran A, Akter S, Ren SJF, Dubey R, Childe SJ (2017) Big data analytics and firm
performance: effects of dynamic capabilities. J Bus Res 70:356–365
Wang N, Liang H, Zhong W, Xue Y, Xiao J (2012) Resource structuring or capability building? An
empirical study of the business value of information technology. J Manag Inf Syst 29(2):325–367
Wernerfelt B (1984) A resource-based view of the firm. Strateg Manag J 5(2):171–180
White C (2011) Using big data for smarter decision making IBM. Yorktown Heights, New York
Winter SG (2003) Understanding dynamic capabilities. Strateg Manag J 24(10):991–995
Wixom BH, Watson HJ, Werner T (2011) Developing an enterprise business intelligence capability: the
Norfolk southern journey. MISQ Exec 10(2):61–71
Wu LY (2007) Entrepreneurial resources, dynamic capabilities and start-up performance of Taiwan’s
high-tech firms. J Bus Res 60(5):549–555
Wu SJ, Melnyk SA, Flynn BB (2010) Operational capabilities: the secret ingredient. Decis Sci
41(4):721–754
Xu P, Kim J (2014) Achieving dynamic capabilities with business intelligence. In: Pacific Asia
conference on information systems PACIS, p 330
Big data analytics capabilities: a systematic literature… 577
123
Xu Z, Frankwick GL, Ramirez E (2016) Effects of big data analytics and traditional marketing analytics
on new product success: a knowledge fusion perspective. J Bus Res 69(5):1562–1566
Zikopoulos P, Eaton C (2011) Understanding big data: analytics for enterprise class Hadoop and
streaming data. McGraw-Hill Osborne Media, New York City
Zollo M, Winter SG (2002) Deliberate learning and the evolution of dynamic capabilities. Organ Sci
13(3):339–351
578 P. Mikalef et al.
123
- Big data analytics capabilities: a systematic literature review and research agenda
- Abstract
- Introduction
- Research methodology
- Protocol development
- Inclusion and exclusion criteria
- Data sources and search strategy
- Quality assessment
- Data extraction and synthesis of findings
- Defining big data in the business context
- Big data
- Big data analytics
- Big data analytics capability
- Toward the development of a big data analytics capability
- Resource based theory
- The dynamic capabilities view of the firm
- Resources of a big data analytics capability
- Tangible resources
- Intangible resources
- Human skills and knowledge
- Areas of big data analytics
- Discussion
- Theme 1: resource orchestration of big data analytics
- Theme 2: decoupling big data analytics capability from big data-enabled capabilities
- Theme 4: turning big data analytics insights into action
- Theme 5: trust of top managers in big data analytics insights
- Theme 6: business value measurement
- Conclusion
- References
© Kitchenham, 2004
Procedures for Performing Systematic Reviews
Barbara Kitchenham
e-mail: [email protected]
Joint Technical Report
Software Engineering Group
Department of Computer Science Keele University
Keele, Staffs ST5 5BG, UK
Keele University Technical Report TR/SE-0401
ISSN:1353-7776
and
Empirical Software Engineering National ICT Australia Ltd.
Bay 15 Locomotive Workshop Australian Technology Park Garden Street, Eversleigh
NSW 1430, Australia
NICTA Technical Report 0400011T.1
July, 2004
i
0. Document Control Section
0.1 Contents
0. Document Control Section......................................................................................i
0.1 Contents ..........................................................................................................i 0.2 Document Version Control .......................................................................... iii 0.3 Executive Summary ......................................................................................iv
1. Introduction............................................................................................................1 2. Systematic Reviews ...............................................................................................1
2.1 Reasons for Performing Systematic Reviews ................................................1 2.2 The Importance of Systematic Reviews ........................................................2 2.3 Advantages and disadvantages ......................................................................2 2.4 Feature of Systematic Reviews ......................................................................2
3. The Review Process ...............................................................................................3 4. Planning .................................................................................................................3
4.1 The need for a systematic review...................................................................3 4.2 Development of a Review Protocol ...............................................................4
4.2.1 The Research Question ..........................................................................5 4.2.1.1 Question Types ..................................................................................5 4.2.1.2 Question Structure .............................................................................6
4.2.1.2.1 Population ....................................................................................6 4.2.1.2.2 Intervention ..................................................................................6 4.2.1.2.3 Outcomes .....................................................................................6 4.2.1.2.4 Experimental designs ...................................................................7
4.2.2 Protocol Review.....................................................................................7 5. Conducting the review ...........................................................................................7
5.1 Identification of Research ..............................................................................7 5.1.1 Generating a search strategy ..................................................................7 5.1.2 Publication Bias .....................................................................................8 5.1.3 Bibliography Management and Document Retrieval ............................9 5.1.4 Documenting the Search ........................................................................9
5.2 Study Selection ..............................................................................................9 5.2.1 Study selection criteria...........................................................................9 5.2.2 Study selection process ........................................................................10 5.2.3 Reliability of inclusion decisions.........................................................10
5.3 Study Quality Assessment ...........................................................................10 5.3.1 Quality Thresholds...............................................................................11 5.3.2 Development of Quality Instruments...................................................15 5.3.3 Using the Quality Instrument...............................................................16 5.3.4 Limitations of Quality Assessment ......................................................16
5.4 Data Extraction ............................................................................................17 5.4.1 Design of Data Extraction Forms ........................................................17 5.4.2 Contents of Data Collection Forms......................................................17 5.4.3 Data extraction procedures ..................................................................17 5.4.4 Multiple publications of the same data ................................................18 5.4.5 Unpublished data, missing data and data requiring manipulation .......18
5.5 Data Synthesis..............................................................................................18
ii
5.5.1 Descriptive synthesis ...........................................................................19 5.5.2 Quantitative Synthesis .........................................................................19 5.5.3 Presentation of Quantitative Results ....................................................20 5.5.4 Sensitivity analysis...............................................................................21 5.5.5 Publication bias ....................................................................................21
6. Reporting the review............................................................................................22 6.1 Structure for systematic review ...................................................................22 6.2 Peer Review .................................................................................................22
7. Final remarks .......................................................................................................25 8. References............................................................................................................25 Appendix 1 Steps in a systematic review ................................................................27
iii
0.2 Document Version Control
Document status
Version Number
Date Changes from previous version
Draft 0.1 1 April 2004 None Published 1.0 29 June 2004 Correction of typos
Additional discussion of problems of assessing evidence Section 7 “Final Remarks” added.
iv
0.3 Executive Summary
The objective of this report is to propose a guideline for systematic reviews appropriate for software engineering researchers, including PhD students. A systematic review is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest. Systematic reviews aim to present a fair evaluation of a research topic by using a trustworthy, rigorous, and auditable methodology. The guideline presented in this report was derived from three existing guidelines used by medical researchers. The guideline has been adapted to reflect the specific problems of software engineering research. The guideline covers three phases of a systematic review: planning the review, conducting the review and reporting the review. It is at a relatively high level. It does not consider the impact of question type on the review procedures, nor does it specify in detail mechanisms needed to undertake meta-analysis.
1
1. Introduction
This document presents a general guideline for undertaking systematic reviews. The goal of this document is to introduce the concept of rigorous reviews of current empirical evidence to the software engineering community. It is aimed at software engineering researchers including PhD students. It does not cover details of meta- analysis (a statistical procedure for synthesising quantitative results from different studies), nor does it discuss the implications that different types of systematic review questions have on systematic review procedures. The document is based on a review of three existing guidelines for systematic reviews: 1. The Cochrane Reviewer’s Handbook [4]. 2. Guidelines prepared by the Australian National Health and Medical Research
Council [1] and [2]. 3. CRD Guidelines for those carrying out or commissioning reviews [12]. In particular the structure of this document owes much to the CRD Guidelines. All these guidelines are intended to aid medical researchers. This document attempts to adapt the medical guidelines to the needs of software engineering researchers. It discusses a number of issues where software engineering research differs from medical research. In particular, software engineering research has relatively little empirical research compared with the large quantities of research available on medical issues, and research methods used by software engineers are not as rigorous as those used by medical researchers. The structure of the report is as follows: 1. Section 2 provides an introduction to systematic reviews as a significant
research method. 2. Section 3 specifies the stages in a systematic review. 3. Section 4 discusses the planning stages of a systematic review 4. Section 5 discusses the stages involved in conducting a systematic review 5. Section 6 discusses reporting a systematic review.
2. Systematic Reviews
A systematic literature review is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest. Individual studies contributing to a systematic review are called primary studies; a systematic review is a form a secondary study.
2.1 Reasons for Performing Systematic Reviews
There are many reasons for undertaking a systematic review. The most common reasons are: • To summarise the existing evidence concerning a treatment or technology e.g. to
summarise the empirical evidence of the benefits and limitations of a specific agile method.
2
• To identify any gaps in current research in order to suggest areas for further investigation.
• To provide a framework/background in order to appropriately position new research activities.
However, systematic reviews can also be undertaken to examine the extent to which empirical evidence supports/contradicts theoretical hypotheses, or even to assist the generation of new hypotheses (see for example [10]).
2.2 The Importance of Systematic Reviews
Most research starts with a literature review of some sort. However, unless a literature review is thorough and fair, it is of little scientific value. This is the main rationale for undertaking systematic reviews. A systematic review synthesises existing work in manner that is fair and seen to be fair. For example, systematic reviews must be undertaken in accordance with a predefined search strategy. The search strategy must allow the completeness of the search to be assessed. In particular, researchers performing a systematic review must make every effort to identify and report research that does not support their preferred research hypothesis as well as identifying and reporting research that supports it.
2.3 Advantages and disadvantages
Systematic reviews require considerably more effort than traditional reviews. Their major advantage is that they provide information about the effects of some phenomenon across a wide range of settings and empirical methods. If studies give consistent results, systematic reviews provide evidence that the phenomenon is robust and transferable. If the studies give inconsistent results, sources of variation can be studied. A second advantage, in the case of quantitative studies, is that it is possible to combine data using meta-analytic techniques. This increases the likelihood of detecting real effects that individual smaller studies are unable to detect. However, increased power can also be a disadvantage, since it is possible to detect small biases as well as true effects.
2.4 Feature of Systematic Reviews
Some of the features that differentiate a systematic review from a conventional literature review are: • Systematic reviews start by defining a review protocol that specifies the research
question being addressed and the methods that will be used to perform the review. • Systematic reviews are based on a defined search strategy that aims to detect as
much of the relevant literature as possible. • Systematic reviews document their search strategy so that readers can access its
rigour and completeness. • Systematic reviews require explicit inclusion and exclusion criteria to assess each
potential primary study. • Systematic reviews specify the information to be obtained from each primary
study including quality criteria by which to evaluate each primary study. • A systematic review is a prerequisite for quantitative meta-analysis
3
3. The Review Process
A systematic review involves several discrete activities. Existing guidelines for systematic reviews have different suggestions about the number and order of activities (see Appendix 1). This documents summarises the stages in a systematic review into three main phases: Planning the Review, Conducting the Review, Reporting the Review. The stages associated with planning the review are: 1. Identification of the need for a review 2. Development of a review protocol. The stages associated with conducting the review are: 1. Identification of research 2. Selection of primary studies 3. Study quality assessment 4. Data extraction & monitoring 5. Data synthesis. Reporting the review is a single stage phase. Each phase is discussed in detail in the following sections. Other activities identified in the guidelines discussed in Appendix 1 are outside the scope of this document. The stages listed above may appear to be sequential, but it is important to recognise that many of the stages involve iteration. In particular, many activities are initiated during the protocol development stage, and refined when the review proper takes place. For example: • The selection of primary studies is governed by inclusion and exclusion criteria.
These criteria are initially specified when the protocol is defined but may be refined after quality criteria are defined.
• Data extraction forms initially prepared during construction of the protocol will be amended when quality criteria are agreed.
• Data synthesis methods defined in the protocol may be amended once data has been collected.
The systematic reviews road map prepared by the Systematic Reviews Group at Berkley demonstrates the iterative nature of the systematic review process very clearly [15].
4. Planning
4.1 The need for a systematic review
The need for a systematic review arises from the requirement of researchers to summarise all existing information about some phenomenon in a thorough and unbiased manner. This may be in order to draw more general conclusion about some phenomenon than is possible from individual studies, or as a prelude to further research activities.
4
Prior to undertaking a systematic review, researchers should ensure that a systematic review is necessary. In particular, researchers should identify and review any existing systematic reviews of the phenomenon of interest against appropriate evaluation criteria. CRC [12] suggests the following checklist: • What are the review’s objectives? • What sources were searched to identify primary studies? Were there any
restrictions? • What were the inclusion/exclusion criteria and how were they applied? • What criteria were used to assess the quality of primary studies and how were
they applied? • How were the data extracted from the primary studies? • How were the data synthesised? How were differences between studies
investigated? How were the data combined? Was it reasonable to combine the studies? Do the conclusions flow from the evidence?
From a more general viewpoint, Greenlaugh [9] suggests the following questions: • Can you find an important clinical question, which the review addressed?
(Clearly, in software engineering, this should be adapted to refer to an important software engineering question.)
• Was a thorough search done of the appropriate databases and were other potentially important sources explored?
• Was methodological quality assessed and the trials weighted accordingly? • How sensitive are the results to the way that the review has been done? • Have numerical results been interpreted with common sense and due regard to the
broader aspects of the problem?
4.2 Development of a Review Protocol
A review protocol specifies the methods that will be used to undertake a specific systematic review. A pre-defined protocol is necessary to reduce the possibility researcher bias. For example, without a protocol, it is possible that the selection of individual studies or the analysis may be driven by researcher expectations. In medicine, review protocols are usually submitted to peer review. The components of a protocol include all the elements of the review plus some additional planning information: • Background. The rationale for the survey. • The research questions that the review is intended answer. • The strategy that will be used to search for primary studies including search terms
and resources to be searched, resources include databases, specific journals, and conference proceedings. An initial scoping study can help determine an appropriate strategy.
• Study selection criteria and procedures. Study selection criteria determine criteria for including in, or excluding a study from, the systematic review. It is usually helpful to pilot the selection criteria on a subset of primary studies. The protocol should describe how the criteria will be applied e.g. how many assessors will evaluate each prospective primary study, and how disagreements among assessors will be resolved.
5
• Study quality assessment checklists and procedures. The researchers should develop quality checklists to assess the individual studies. The purpose of the quality assessment will guide the development of checklists.
• Data extraction strategy. This should define how the information required from each primary study would be obtained. If the data require manipulation or assumptions and inferences to be made, the protocol should specify an appropriate validation process.
• Synthesis of the extracted data. This should define the synthesis strategy. This should clarify whether or not a formal meta-analysis is intended and if so what techniques will be used.
• Project timetable. This should define the review plan.
4.2.1 The Research Question
4.2.1.1 Question Types The most important activity during protocol is to formulate the research question. The Australian NHMR Guidelines [1] identify six types of health care questions that can be addressed by systematic reviews: 1. Assessing the effect of intervention. 2. Assessing the frequency or rate of a condition or disease. 3. Determining the performance of a diagnostic test. 4. Identifying aetiology and risk factors. 5. Identifying whether a condition can be predicted. 6. Assessing the economic value of an intervention or procedure. In software engineering, it is not clear what the equivalent of a diagnostic test would be, but the other questions can be adapted to software engineering issues as follows: • Assessing the effect of a software engineering technology. • Assessing the frequency or rate of a project development factor such as the
adoption of a technology, or the frequency or rate of project success or failure. • Identifying cost and risk factors associated with a technology. • Identifying the impact of technologies on reliability, performance and cost
models. • Cost benefit analysis of software technologies. Medical guidelines often provide different guidelines and procedures for different types of question. This document does not go to this level of detail. The critical issue in any systematic review is to ask the right question. In this context, the right question is usually one that: • Is meaningful and important to practitioners as well as researchers. For example,
researchers might be interested in whether a specific analysis technique leads to a significantly more accurate estimate of remaining defects after design inspections. However, a practitioner might want to know whether adopting a specific analysis technique to predict remaining defects is more effective than expert opinion at identifying design documents that require re-inspection.
• Will lead either to changes in current software engineering practice or to increased confidence in the value of current practice. For example, researchers
6
and practitioners would like to know under what conditions a project can safely adopt agile technologies and under what conditions it should not.
• Identify discrepancies between commonly held beliefs and reality. Nonetheless, there are systematic reviews that ask questions that are primarily of interest to researchers. Such reviews ask questions that identify and/or scope future research activities. For example, a systematic review in a PhD thesis should identify the existing basis for the research student’s work and make it clear where the proposed research fits into the current body of knowledge.
4.2.1.2 Question Structure Medical guidelines recommend considering a question from three viewpoints: • The population, i.e. the people affected by the intervention. • The interventions usually a comparison between two or more alternative
treatments. • The outcomes, i.e. the clinical and economic factors that will be used to compare
the interventions. In addition, study designs appropriate to answering the review questions may be identified.
4.2.1.2.1 Population In software engineering experiments, the populations might be any of the following: • A specific software engineering role e.g. testers, managers. • A type of software engineer, e.g. a novice or experienced engineer. • An application area e.g. IT systems, command and control systems. A question may refer to very specific population groups e.g. novice testers, or experienced software architects working on IT systems. In medicine the populations are defined in order to reduce the number of prospective primary studies. In software engineering far less primary studies are undertaken, thus, we may need to avoid any restriction on the population until we come to consider the practical implications of the systematic review.
4.2.1.2.2 Intervention Interventions will be software technologies that address specific issues, for example, technologies to perform specific tasks such as requirements specification, system testing, or software cost estimation.
4.2.1.2.3 Outcomes Outcomes should relate to factors of importance to practitioners such as improved reliability, reduced production costs, and reduced time to market. All relevant outcomes should be specified. For example, in some cases we require interventions that improve some aspect of software production without affecting another e.g. improved reliability with no increase in cost. A particular problem for software engineering experiments is the use of surrogate measures for example, defects found during system testing as a surrogate for quality,
7
or coupling measures for design quality. Studies that use surrogate measures may be misleading and conclusions based on such studies may be less robust.
4.2.1.2.4 Experimental designs In medical studies, researches may be able to restrict systematic reviews to primary of studies of one particular type. For example, Cochrane reviews are usually restricted to randomised controlled trials (RCTs). In other circumstances, the nature of the question and the central issue being addressed may suggest that certain studies design are more appropriate than others. However, this approach can only be taken in a discipline where the amount of available research is a major problem. In software engineering, the paucity of primary studies is more likely to be the problem for systematic reviews and we are more likely to need protocols for aggregating information from studies of widely different types. A starting point for such aggregation is the ranking of primary studies of different types; this is discussed in Section 5.3.1.
4.2.2 Protocol Review The protocol is a critical element of any systematic review. Researchers must agree a procedure for reviewing the protocol. If appropriate funding is available, a group of independent experts should be asked to review the protocol. The same experts can later be asked to review the final report. PhD students should present their protocol to their supervisors for review and criticism.
5. Conducting the review
Once the protocol has been agreed, the review proper can start. This involves: 1. Identification of research 2. Selection of studies 3. Study quality assessment 4. Data extraction and monitoring progress 5. Data synthesis Each of these stages will be discussed in this section. Although some stages must proceed sequentially, some stages can be undertaken simultaneously.
5.1 Identification of Research
The aim of a systematic review is to find as many primary studies relating to the research question as possible using an unbiased search strategy. For example, it is necessary to avoid language bias. The rigour of the search process is one factor that distinguishes systematic reviews from traditional reviews.
5.1.1 Generating a search strategy It is necessary to determine and follow a search strategy. This should be developed in consultation with librarians. Search strategies are usually iterative and benefit from: • Preliminary searches aimed at both identifying existing systematic reviews and
assessing the volume of potentially relevant studies.
8
• Trial searchers using various combinations of search terms derived from the research question
• Reviews of research results • Consultations with experts in the field A general approach is to break down the question into individual facets i.e. population, intervention, outcomes, study designs. Then draw up a list of synonyms, abbreviations, and alternative spellings. Other terms can be obtained by considering subject headings used in journals and data bases. Sophisticated search strings can then be constructed using Boolean AND’s and OR’s. Initial searches for primary studies can be undertaken initially using electronic databases but this is not sufficient. Other sources of evidence must also be searched (sometimes manually) including: • Reference lists from relevant primary studies and review articles • Journals (including company journals such as the IBM Journal of Research and
Development), grey literature (i.e. technical reports, work in progress) and conference proceedings
• Research registers • The Internet. It is also important to identify specific researchers to approach directly for advice on appropriate source material. Medical researchers have developed pre-packaged research strategies. Software Engineering Researchers need to develop and publish such strategies including identification of relevant electronic databases.
5.1.2 Publication Bias Publication bias refers to the problem that positive results are more likely to be published than negative results. The concept of positive or negative results sometimes depends on the viewpoint of the researcher. (For example, evidence that full mastectomies were not always required for breast cancer was actually an extremely positive result for breast cancer sufferers). However, publication bias remains a problem particularly for formal experiments, where failure to reject the null hypothesis is considered less interesting than an experiment that is able to reject the null hypothesis. Publication bias can lead to systematic bias in systematic reviews unless special efforts are made to address this problem. Many of the standard search strategies identified above are used to address this issue including: • Scanning the grey literature • Scanning conference proceedings • Contacting experts and researches working in the area and asking them if they
know of any unpublished results. In addition, statistical analysis techniques can be used to identify the potential significance of publication bias (se Section 5.5.5).
9
5.1.3 Bibliography Management and Document Retrieval Bibliographic packages such as Reference Manager or Endnote are very useful to manage the large number of reference that can be obtained from a thorough literature research. Once reference lists have been finalised the full articles of potentially useful studies will need to be obtained. A logging system is needed to make sure all relevant studies are obtained.
5.1.4 Documenting the Search The process of performing a systematic review must be transparent and replicable: • The review must be documented in sufficient detail for readers to be able to
assess the thoroughness of the search. • The search should be documented as it occurs and changes noted and justified. • The unfiltered search results should be saved and retained for possible reanalysis. Procedures for documenting the search process are given in Table 1.
Table 1 Search process documentation Data Source Documentation Electronic database Name of database
Search strategy for each database Date of search Years covered by search
Journal Hand Searches Name of journal Years searched Any issues not searched
Conference proceedings Title of proceedings Name of conference (if different) Title translation (if necessary) Journal name (if published as part of a journal)
Efforts to identify unpublished studies
Research groups and researchers contacted (Names and contact details) Research web sites searched (Date and URL)
Other sources Date Searched/Contacted URL Any specific conditions pertaining to the search
5.2 Study Selection
Once the potentially relevant primary studies have been obtained, they need to be assessed for their actual relevance.
5.2.1 Study selection criteria Study selection criteria are intended to identify those primary studies that provide direct evidence about the research question. In order to reduce the likelihood of bias, selection criteria should be decided during the protocol definition. Inclusion and exclusion criteria should be based on the research question. They should be piloted to ensure that they can be reliably interpreted and that they classify studies correctly.
10
Issues: • It is important to avoid, as far as possible, exclusions based on the language of the
primary study. It is often possible to cope with French or German abstracts, but Japanese or Chinese papers are often difficult to access unless they have a well- structured English abstract.
• It is possible that inclusion decisions could be affected by knowledge of the authors, institutions, journals or year of publication. Some medical researchers have suggested reviews should be done after such information has been removed. However, it takes time to do this and experimental evidence suggests that masking the origin of primary studies does not improve reviews [3].
5.2.2 Study selection process Study selection is a multistage process. Initially, selection criteria should be interpreted liberally, so that unless studies identified by the electronic and hand searchers can be clearly excluded based on titles and abstracts, full copies should be obtained. Final inclusion/exclusion decisions should be made after the full texts have been retrieved. It is useful to maintain a list of excluded studies identifying the reason for exclusion.
5.2.3 Reliability of inclusion decisions When two or more researchers assess each paper, agreement between researchers can be measured using the Cohen Kappa statistic [6]. Each disagreement must be discussed and resolved. This may be a matter of referring back to the protocol or may involve writing to the authors for additional information. Uncertainty about the inclusion/exclusion of some studies should be investigated by sensitivity analysis. A single researcher should consider discussing included and excluded papers with an expert panel.
5.3 Study Quality Assessment
In addition, to general inclusion exclusion criteria, it is generally considered important to assess the “quality” of primary studies: • To provide still more detailed inclusion/exclusion criteria. • To investigate whether quality differences provide an explanation for differences
in study results. • As a means of weighting the importance of individual studies when results are
being synthesised. • To guide the interpretation of findings and determine the strength of inferences. • To guide recommendations for further research. An initial difficulty is that there is no agreed definition of study “quality”. However, the CRD Guidelines [12] and the Cochrane Reviewers’ Handbook [4] both suggest that quality relates to the extent to which the study minimises bias and maximises internal and external validity (see Table 2).
11
Table 2 Quality concept definitions Term Synonyms Definition Bias Systematic error A tendency to produce results that depart systematically
from the ‘true’ results. Unbiased results are internally valid Internal validity Validity The extent to which the design and conduct of the study are
likely to prevent systematic error. Internal validity is a prerequisite for external validity.
External validity Generalisability, Applicability
The extent to which the effects observed in the study are applicable outside of the study.
5.3.1 Quality Thresholds The CRD Guideline [4] suggests using an assessment of study design to guarantee a minimum level of quality. The Australian National Health and Medical Research Council guidelines [2] suggest that study design is considered during assessment of evidence rather than during the appraisal and selection of studies. Both groups however suggest a hierarchy of study designs (see Table 3 and Table 4).
Table 3 CRD Hierarchy of evidence Level Description 1 Experimental studies (i.e. RCT with concealed allocation) 2 Quasi-experimental studies (i.e. studies without randomisation) 3 Controlled observational studies 3a Cohort studies 3b Case control studies 4 Observational studies without control groups 5 Expert opinion based on theory, laboratory research or consensus
Table 4 Australian NHMRC Study design hierarchy Level I Evidence obtained from a systematic review of all relevant randomised trials Level II Evidence obtained from at least one properly-designed randomised controlled trial Level III-1 Evidence obtained from well-designed pseudo-randomised controlled trials (i.e. non-
random allocation to treatment) Level III-2 Evidence obtained from comparative studies with concurrent controls and allocation
not randomised, cohort studies, case-control studies or interrupted time series with a control group.
Level III-3 Evidence obtained from comparative studies with historical control, two or more single arm studies, or interrupted time series without a parallel control group
Level IV Evidence obtained from case series, either post-test or pretest/post-test In order to understand Table 3 and Table 4 some additional definitions of studies types is given in Table 5, where experimental studies are those in which some conditions, particularly those concerning the allocation of participants to different treatment groups are under the control of investigator and observational studies are those in which uncontrolled variation in treatment or exposure among study participants is investigated. Although the definitions given in Table 5 appear appropriate to software engineering studies (replacing the word disease with condition), it is important to note one critical difference between medical experiments and software engineering experiments. Most
12
experiments performed in academic settings cannot be equated to randomised controlled trials (RCTs) in medicine.
Table 5 Definition of study designs Design Type
Synonym Basic Type Definition Source
Randomised Controlled Trial (RCT)
Randomised Clinical Trial
Experiment An experiment in which investigators randomly allocate eligible people into intervention groups
[5]
Quasi- randomised trial
Pseudo- randomised controlled trial
Experiment A study in which the allocation of participants to different intervention groups is controlled by the investigator but the method falls short of genuine randomisation and allocation concealment.
[12]
Cohort study
Follow-up study, incidence study, longitudinal study, prospective study
Observation An observational study in which a defined group of people (the cohort) is followed over time. The outcomes of people in subsets are compared to examine for example people who were exposed to or not exposed (or exposed at different levels) to a particular intervention.
[12]
Concurrent cohort study
Observation A study where a cohort is assembled in the present and followed into the future
[12]
Historical cohort study
Observation A study where a cohort is identified from past records and followed from that time to the present.
[12]
Case-control study
Observation Subjects with the outcome or disease and an appropriate group of controls without the outcome or disease are selected and information is obtained about the previous exposure to the treatment or other factor being studied
[2]
Historical control
Observation Outcomes for a prospectively collected group of subjects exposed to a new treatment/intervention are compared with either a previously published series or previously treated subjects at the same institutions.
[2]
Interrupted time series
Observation Trends in the outcomes or diseases are compared over multiple time points before and after introduction of the treatment/intervention or other factor being studied.
[2]
Cross- sectional study
Observation Examination of relationships between diseases and other variables of interest as they exist in a defined population at one particular time
[12]
Case series Observation A group of subjects are exposed to the treatment or intervention
[2]
Post-test case series
Observation A case series where only outcomes after the intervention are recorded in the case series, so no comparisons can be made.
[2]
Pre-test / post-test case series
Before-and- after study
Observation A case series where outcomes are measured in subjects before and after exposure to the treatment/intervention for comparison.
[2]
13
RCTs involve real patients with real diseases receiving a new treatment to manage their condition. That is, RCTs are trials of treatment under its actual use conditions. The majority of academic experiments involve students doing constrained tasks in artificial environments. Thus, the major issue for software engineering study hierarchies is whether small-scale experiments are considered the equivalent of laboratory experiments and evaluated at the lowest level of evidence, or whether they should be ranked higher. In my opinion, they should be ranked higher than expert opinion. I would consider them equivalent in value to case series or observational studies without controls. Two other issues that need to be resolved are: • Whether or not systematic reviews are included in the hierarchy. • Whether or not expert opinion is included in the hierarchy. The inclusion of systematic reviews depends on whether you are classifying individual studies or assessing the level of evidence. For assessing individual primary studies, systematic reviews are, of course, excluded. For assessing the level of evidence, systematic reviews should be considered the highest level of evidence. However, in contrast to the implication in the Australian Hierarchy in Table 4, I believe software engineers must consider systematic reviews of many types of primary study not only randomised controlled trials. The Australian NHMRC guidelines [2] do not included expert opinion in their hierarchy. The authors remark that the exclusion is a result of studies identifying the fallibility of expert opinion. In software engineering we may have little empirical evidence, so may have to rely more on expert opinion than medical researchers. However, we need to recognise the weakness of such evidence.
Table 6 Study design hierarchy for Software Engineering 1 Evidence obtained from at least one properly-designed randomised controlled trial 2 Evidence obtained from well-designed pseudo-randomised controlled trials (i.e. non-
random allocation to treatment) 3-1 Evidence obtained from comparative studies with concurrent controls and allocation
not randomised, cohort studies, case-control studies or interrupted time series with a control group.
3-2 Evidence obtained from comparative studies with historical control, two or more single arm studies, or interrupted time series without a parallel control group
4-1 Evidence obtained from a randomised experiment performed in an artificial setting 4-2 Evidence obtained from case series, either post-test or pre-test/post-test 4-3 Evidence obtained from a quasi-random experiment performed in an artificial setting 5 Evidence obtained from expert opinion based on theory or consensus These considerations lead to the hierarchy shown in Table 6 for Software Engineering. Studies. This table includes reference to randomised controlled trials although I am aware of only one software engineering experiment that comes anywhere close to a randomised controlled trial in the sense that it undertakes an experiment in a real-life situation [11]. In this study, Jørgensen and Carelius requested a bid for a real project from a large number of commercial software companies in Norway. Companies were selected using stratified random sampling. Once the full sample was obtained, companies were randomly assigned to two groups. One group of companies were involved in pre-study phase and the bidding phase, the other companies were only involved in the bidding phase. The treatment in this case, was the pre-study activity, which involved companies providing an initial non-binding
14
preliminary bid. One aspect that is not consistent with an RCT is that companies were paid for their time in order to compensate them for providing additional information to the experimenters. In addition, the study was not aimed at formal hypothesis testing, so the outcome was a possible explanatory theory rather than a statement of expected treatment effect. Normally, primary study hierarchies are used to set a minimum requirement on the type of study included in the systematic review. In software engineering, we will usually accept all levels of evidence. The only threshold that might be viable would be to exclude level 5 evidence when there are a reasonable number of primary studies at a greater level (where a reasonable number must be decided by the researchers, but should be more than 2). Categorising evidence hierarchies does not by itself solve the problem of how to accumulate evidence from studies in different categories. We discuss some fairly simple ideas in Section 5.5.4 used to present evidence, but we may need to identify new methods of accumulating evidence from different types of study. For example, Hardman and Ayton discuss a system to allow the accumulation of qualitative as well as quantitative evidence in the form arguments that are for or against proposition [13]. In addition, we need better understand the strength of evidence from different types of study. However, this is difficult. For example, there is no agreement among medical practitioners of the extent to which results from observational studies can really be trusted. Some medical researchers are critical of the reliance on RCTs and report cases where observational studies produced almost identical results to RCTs [8]. Concato and Horowitz suggest that improvements in reporting clinical conditions (i.e. collecting more information about individual patients and the reasons for assigning the patient to a particular treatment) would make observational studies as reliable as RCTs [7]. In contrast, Lawlor et al. discuss an example where results of a RCT proved that observational studies were incorrect [14]. Specifically, beneficial effects of vitamins in giving protection against heart disease found in two observational studies could not be detected in a randomised controlled trial. They suggest that better identification and adjustment for possible confounding factors would improve the reliability of observational studies. In addition, Vandenbroucke suggests that observational studies are appropriate for detecting negative side-effects, but not positive side-effects of treatments [17]. Observational studies and experiments in software engineering often have more in common with studies in the social sciences than medicine. For example, both social science and software engineering struggle with the problems both of defining and measuring constructs of interest, and of understanding the impact of experimental context on study results. From the viewpoint of social science, Shadish et al. provide a useful discussion of the study design and analysis methods that can improve the validity of experiments and quasi-experiments [16]. They emphasise the importance of identifying and either measuring or controlling confounding factors. They also discuss threats to validity across all elements of a study i.e. subjects, treatments, observations and settings.
15
5.3.2 Development of Quality Instruments Once the primary studies have been selected a more detailed quality assessment needs to be made. This allows researchers to assess differences in the executions of studies within design categories. This information is important for data synthesis and interpretation of results. Detailed quality assessments are usually based on “quality instruments” which are checklists of factors that need to be assessed for each study. If quality items within a checklist are assigned numerical scales numerical assessments of quality can be obtained. Checklists are usually derived from a consideration of factors that could bias study results. The CRD Guidelines [12], the Australian National Health and Medical Research Council Guidelines [1], and the Cochrane Reviewers’ Handbook [4] all refer to four types of bias shown in Table 7. (I have amended the definitions (slightly) and protection mechanisms (considerably) to address software engineering rather than medicine.) In particular, medical researchers rely on “blinding” subjects and experimenters (i.e. making sure that neither the subject nor the researcher knows which treatment a subject is assigned to) to address performance and measurement bias. However, that protocol is often impossible for software engineering experiments.
Table 7 Types of Bias Type Synonyms Definition Protection mechanism Selection bias
Allocation bias
Systematic difference between comparison groups with respect to treatment
Randomisation of a large number of subjects with concealment of the allocation method (e.g. allocation by computer program not experimenter choice).
Performance bias
Systematic difference is the conduct of comparison groups apart from the treatment being evaluated.
Replication of the studies using different experimenters. Use of experimenters with no personal interest in either treatment.
Measurement bias
Detection Bias
Systematic difference between the groups in how outcomes are ascertained.
Blinding outcome assessors to the treatments is sometimes possible.
Attrition bias Exclusion bias
Systematic differences between comparison groups in terms of withdrawals or exclusions of participants from the study sample.
Reporting of the reasons for all withdrawals. Sensitivity analysis including all excluded participants.
The factors identified in Table 7 are refined into a quality instrument by considering: • Generic items that relate to features of particular study designs such as lack of
appropriate blinding, unreliable measurement techniques, inappropriate selection of subjects, and inappropriate statistical analysis.
• Specific items that relate to the review’s subject area such as use of outcome measures inappropriate for answering the research question.
More detailed discussion of bias (or threats to validity) from the viewpoint of the social sciences rather than medicine can be found in Shadish et al. [16].
16
Examples of generic quality criteria for several types of study design are shown in Table 8. The items were derived from lists in [2] and [12]. If required, researchers may construct a measurement scale for each item. Whatever form the quality instrument takes, it should be assessed for reliability and usability in a pilot project before being applied to all the selected studies.
5.3.3 Using the Quality Instrument Quality appraisal of each primary study allows researchers to group studies by quality prior to any synthesis of results. Researchers can then investigate whether there are systematic differences between primary studies in different quality groups. Some researchers have suggested weighting results using quality scores. This idea is not recommended by any of the medical guidelines.
5.3.4 Limitations of Quality Assessment Primary studies are often poorly reported, so it may not be possible to determine how to assess a quality criterion. It is possible to assume that because something wasn’t reported, it wasn’t done. This assumption may be incorrect. Researchers should attempt to obtain more information from the authors of the study.
Table 8 Example of Quality Criteria Study type Quality criteria Cohort studies How were subjects chosen for the new intervention? How were subjects selected for the comparison or control? Were drop-out rates and reasons for drop-out similar across intervention and
unexposed groups? Does the study adequately control for demographic characteristics, and other
potential confounding variables in the design or analysis? Was the measurement of outcomes unbiased (i.e. blinded to treatment group and
comparable across groups)? Were there exclusions from the analysis? Case-control studies
How were cases defined and selected?
How were controls defined and selected? (I.e. were they randomly selected from the source population of the cases)
How comparable are the cases and the controls with respect to potential confounding factors?
Does the study adequately control for demographic characteristics, and other potential confounding variables in the design or analysis?
Was measurement of the exposure to the factor of interest adequate and kept blinded to the case/control status?
Were all selected subjects included in the analysis? Were interventions and other exposures assessed in the same way for cases and
controls? Was an appropriate statistical analysis used (i.e. matched or unmatched)? Case series Is the study based on a representative sample from a relevant population? Are criteria for inclusion explicit? Were outcomes assessed using objective criteria? There is limited evidence of relationships between factors that are thought to affect validity and actual study outcomes. Evidence suggests that inadequate concealment of
17
allocation and lack of double-blinding result in over-estimates of treatment effects, but the impact of other quality factors is not supported by empirical evidence. It is possible to identify inadequate or inappropriate statistical analysis, but without access to the original data it is not possible to correct the analysis. Very often software data is confidential and cannot therefore be made available to researchers. In some cases, software engineers may refuse to make their data available to other researchers because they want to continue publishing analyses of the data.
5.4 Data Extraction
The objective of this stage is to design data extraction forms to accurately record the information researchers obtain from the primary studies. To reduce the opportunity for bias, data extraction forms should be defined and piloted when the study protocol is defined.
5.4.1 Design of Data Extraction Forms The data extraction forms must be designed to collect all the information needed to address the review questions and the study quality criteria. They must also collect all data items specified in the review synthesis strategy section of the protocol. In most cases, data extraction will define a set of numerical values that should be extracted for each study (e.g. number of subjects, treatment effect, confidence intervals, etc.). Numerical data are important for any attempt to summarise the results of a set of primary studies and are a prerequisite for meta-analysis (i.e. statistical techniques aimed at integrating the results of the primary studies). Data extraction forms need to be piloted on a sample of primary studies. If several researchers will use the forms, several researchers should take part in the pilot. The pilot studies are intended to assess both technical issues such as the completeness of the forms and usability issues such as the clarity of user instructions and the ordering of questions. Electronic forms are useful and can facilitate subsequent analysis.
5.4.2 Contents of Data Collection Forms In addition, to including all the questions needed to answer the review question and quality evaluation criteria, data collection forms should provide standard information including: • Name of Review • Date of Data extraction • Title, authors, journal, publication details • Space for additional notes
5.4.3 Data extraction procedures Whenever feasible, data extraction should be performed independently by two or more researchers. Data from the researchers must be compared and disagreements resolved either by consensus among researchers or arbitration by an additional independent researcher. Uncertainties about any primary sources for which agreement
18
cannot be reached should be investigated as part of any sensitively analyses. A separate form must be used to mark and correct errors or disagreements. If several researchers each review different primary studies because time or resource constraints prevent all primary papers being assessed by at least two researchers, it is important to ensure employ some method of checking that researchers extract data in a consistent manner. For example, some papers should be reviewed by all researchers (e.g. a random sample of primary studies), so that inter-researcher consistency can be assessed. For single researchers such as PhD students, other checking techniques must be used, for example supervisors should be asked to perform data extraction on a random sample of the primary studies and results cross-checked with those of the student.
5.4.4 Multiple publications of the same data It is important to avoid including multiple publications of the same data in a systematic review synthesis because duplicate reports would seriously bias any results. It may be necessary to contact the authors to confirm whether or not reports refer to the same study. When there are duplicate publications, the most recent should be used.
5.4.5 Unpublished data, missing data and data requiring manipulation
If information is available from studies in progress, it should be included providing appropriate quality information about the study can be obtained and written permission is available from the researchers. Reports do not always include all relevant data. They may also be poorly written and ambiguous. Again the authors should be contacted to obtain the required information. Sometimes primary studies do not provide all the data but it is possible to recreate the required data by manipulating the published data. If any such manipulations are required, data should first be reported in the way they were reported. Data obtained by manipulation should be subject to sensitivity analysis.
5.5 Data Synthesis
Data synthesis involves collating and summarising the results of the included primary studies. Synthesis can be descriptive (non-quantitative). However, it is sometimes possible to complement a descriptive synthesis with a quantitative summary. Using statistical techniques to obtain a quantitative synthesis is referred to as meta-analysis. Description of meta-analysis methods is beyond the scope of this document, although techniques for displaying quantitative results will be described. (To learn more about meta-analysis see [4].) The data synthesis activities should be specified in the review protocol. However, some issues cannot be resolved until the data is actually analysed, for example, subset analysis to investigate heterogeneity is not required if the results show no evidence of heterogeneity.
19
5.5.1 Descriptive synthesis Extracted information about the studies (i.e. intervention, population, context, sample sizes, outcomes, study quality) should be tabulated in a manner consistent with the review question. Tables should be structured to highlight similarities and difference between study outcomes. It is important to identify whether results from studies are consistent one with another (i.e. homogeneous) or inconsistent (e.g. heterogeneous). Results may be tabulated to display the impact of potential sources of heterogeneity, e.g. study type, study quality, and sample size. Quantitative data should also be presented in tabular form including: • Sample size for each intervention • Estimates effect size for each intervention with standard errors for each effect • Difference between the mean values for each intervention, and the confidence
interval for the difference. • Units used for measuring the effect.
5.5.2 Quantitative Synthesis To synthesis quantitative results from different studies, study outcomes must be presented in a comparable way. Medical guidelines suggest different effect measures for different types of outcome. Binary outcomes (Yes/No, Success/Failure) can be measured in several different ways: • Odds. The ratio of the number of subjects in a group with an event to the number
without an event. Thus if 20 projects in a group of 100 project failed to achieve budgetary targets, the odds would be 20/80 or 0.25.
• Risk (proportion, probability, rate) The proportion of subjects in a group observed to have an event. Thus, if 20 out of 100 projects failed to achieve budgetary targets, the risk would be 20/100 or 0.20.
• Odds ratio (OR). The ratio of the odds of an event in the experimental (or intervention) group to the odds of an event on the control group. An OR equal to one indicates no difference between the control and the intervention group. For undesirable outcomes a value less than one indicates that the intervention was successful in reducing risk, for a desirable outcome a value greater than one indicates that the intervention was successful in reducing risk.
• Relative risk (RR) (risk ratio, rate ratio). The ratio of risk in the intervention group to the risk in the control group. An RR of one indicates no difference between comparison groups. For undesirable events an RR less than one indicates the intervention was successful, for desirable events an RR greater than one indicates the intervention was successful.
• Absolute risk reduction (ARR) (risk difference, rate difference). The absolute difference in the event rate between the comparison groups. A difference of zero indicates no difference between the groups. For an undesirable outcome an ARR less than zero indicates a successful intervention, for a desirable outcome an ARR greater than zero indicates a successful intervention.
20
Each of these measures has advantages and disadvantages. For example, odds and odds ratios are criticised for not being well-understood by non-statisticians (other than gamblers), whereas risk measures are generally easier to understand. Alternatively statisticians prefer odd ratios because they have some mathematically desirable properties. Another issue is the relative measures are generally more consistent than absolute measures for statistical analysis, but decision makers need absolute values in order to assess the real benefit of an intervention. Effect measures for continuous data include: • Mean difference. The difference between the means of each group (control and
intervention group). • Weighted mean difference (WMD). When studies have measured the difference
on the same scale, the weight give to each study is usually the inverse of the study variance
• Standardised mean difference (SMD). A common problem when summarising outcomes is that outcomes are often measured in different ways, for example, productivity might be measured in function points per hour, or lines of code per day. Quality might be measured as the probability of exhibiting one or more faults or the number of faults observed. When studies use different scales, the mean difference may be divided by an estimate of the within-groups standard deviation to produce a standardised value without any units. However, SMDs are only valid if the difference in the standard deviations reflect differences in the measurement scale, not real differences among trial populations.
5.5.3 Presentation of Quantitative Results The most common mechanism for presenting quantitative results is a forest plot, as shown in Figure 1. A forest plot presents the means and variance for the difference for each study. The line represents the standard error of the difference, the box represents the mean difference and its size is proportional to the number of subjects in the study. A forest plot may also be annotated with the numerical information indicating the number of subjects in each group, the mean difference and the confidence interval on the mean. If a formal meta-analysis is undertaken, the bottom entry in a forest plot will be the summary estimate of the treatment difference and confidence interval for the summary difference. Figure 1 represents the ideal result of a quantitative summary, the results of the studies basically agree. There is clearly a genuine treatment effect and a single overall summary statistics would be a good estimate of that effect. If effects were very different from study to study, our results would suggest heterogeneity. A single overall summary statistics would probably be of little value. The systematic review should continue with an investigation of the reasons for heterogeneity. To avoid the problems of post-hoc analysis, researchers should identify possible sources of heterogeneity when they construct the review protocol.
21
Figure 1 Example of a forest plot
Study 1
Study 2
Study 3
-0.2 -0.1 0 0.1 0.2 Favours control Favours intervention
5.5.4 Sensitivity analysis Sensitivity analysis is much more important when a full meta-analysis is performed than when no formal meta-analysis is performed. Meta-analysis is used to provide an overall estimate of the treatment effect and its variability. In such cases, the results of the analysis should be repeated on various subsets of primary studies to determine whether the results are robust. The types of subsets selected would be: • High quality primary studies only. • Primary studies of particular types. • Primary studies for which data extraction presented no difficulties (i.e. excluding
any studies where there was some residual disagreement about the data extracted). When a formal meta-analysis is not undertaken, forest plots can be annotated to identify high quality primary studies, the studies can be presented in decreasing order of quality or in decreasing study type hierarchy order. Primary studies where there are queries about the data extracted can also be explicitly identified on the forest plot, by for example, using grey colouring for less reliability studies and black colouring for reliable studies.
5.5.5 Publication bias Funnel plots are used to assess whether or not a systematic review is likely to be vulnerable to publication bias. Funnel plots plot the treatment effect (i.e. mean difference between intervention group and control) against the inverse of the variance or the sample size. A systematic review that exhibited the funnel shape shown in Figure 2 would be assumed not to be exhibiting evidence of publication bias. It would be consistent with studies based on small samples showing more variability in outcome than studies based on large samples. If, however, the points shown as filled- in black dots were not present, the plot would be asymmetric and it would suggest the presence of publication bias. This would suggest the results of the systematic survey must be treated with caution.
22
Figure 2 An example of a funnel plot
1/variance
Treatment effect
6. Reporting the review
It is important to communicate the results of a systematic review effectively. Usually systematic reviews will be reported in at least two formats: 1. In a technical report or in a section of a PhD thesis. 2. In a journal or conference paper. A journal or conference paper will normally have a size restriction. In order to ensure that readers are able to properly evaluate the rigour and validity of a systematic review, journal papers should reference a technical report or thesis that contains all the details. In addition, systematic reviews with important practical results may be summarised in non-technical articles in practitioner magazines, in press releases and in Web pages.
6.1 Structure for systematic review
The structure and contents of reports suggested in [12] is presented in Table 9. This structure is appropriate for technical reports and journals. For PhD theses, the entries marked with an asterisk are not likely to be relevant.
6.2 Peer Review
Journal articles will be peer reviewed as a matter of course. Experts review PhD theses as part of the examination process. In contrast, technical reports are not usually subjected to peer review. However, if systematic reviews are made available on the Web so that results are made available quickly to researchers and practitioners, it is worth organising a peer review. If an expert panel were assembled to review the study protocol, the same panel would be appropriate to undertake peer review of the systematic review report. .
23
Table 9 Structure and contents of reports of systematic reviews Section Subsection Scope Comments Title* The title should be short but informative. It should be based on the question
being asked. In journal papers, it should indicate that the study is a systematic review.
Authorship* When research is done collaboratively, criteria for determining both who should be credited as an author, and the order of author’s names should be defined in advance. The contribution of workers not credited as authors should be noted in the Acknowledgements section.
Context
The importance of the research questions addressed by the review
Objectives
The questions addressed by the systematic review
Methods Data Sources, Study selection, Quality Assessment and Data extraction
Results Main finding including any meta- analysis results and sensitivity analyses.
Executive summary or Structured Abstract*
Conclusions Implications for practice and future research
A structured summary or abstract allows readers to assess quickly the relevance, quality and generality of a systematic review.
Background Justification of the need for the review. Summary of previous reviews
Description of the software engineering technique being investigated and its potential importance
Review questions Each review question should be specified
Identify primary and secondary review questions. Note this section may be included in the background section.
Data sources and search strategy
Study selection Study quality assessment Data extraction
Review Methods
Data synthesis
This should be based on the research protocol. Any changes to the original protocol should be reported.
Included and excluded studies
Inclusion and exclusion criteria List of excluded studies with rationale for exclusion
Study inclusion and exclusion criteria can sometimes best be represented as a flow diagram because studies will be excluded at different stages in the review for different reasons.
24
Findings Description of primary studies Results of any quantitative summaries Details of any meta-analysis
Results
Sensitivity analysis
Non-quantitative summaries should be provided to summarise each of the studies and presented in tabular form. Quantitative summary results should be presented in tables and graphs
Discussion Principal findings These must correspond to the findings discussed in the results section Strengths and Weaknesses Strength and weaknesses of the
evidence included in the review Relation to other reviews, particularly considering any differences in quality and results.
A discussion of the validity of the evidence considering bias in the systematic review allows a reader to assess the reliance that may be placed on the collected evidence.
Meaning of findings Direction and magnitude of effect observed in summarised studies Applicability (generalisability) of the findings
Make clear to what extent the result imply causality by discussing the level of evidence. Discuss all benefits, adverse effects and risks. Discuss variations in effects and their reasons (for example are the treatment effects larger on larger projects).
Practical implications for software development
What are the implications of the results for practitioners? Conclusions Recommendations
Unanswered questions and implications for future research
Acknowledgements* All persons who contributed to the research but did fulfil authorship criteria
Conflict of Interest Any secondary interest on the part of the researchers (e.g. a financial interest in the technology being evaluated) should be declared.
References and Appendices
Appendices can be used to list studies included and excluded from the study, to document search strategy details, and to list raw data from the included studies.
25
7. Final remarks
This report has presented a set of guidelines for planning conducting and reporting systematic review. The guidelines are based on guidelines used in medical research. However, it is important to recognise that software engineering research is not the same as medical research. We do not undertake randomised clinical trials, nor can we use blinding as a means to reduce distortions due to experimenter and subject expectations. Thus, software engineering research studies usually provide only weak evidence compared with RCTs. We need to consider mechanisms to aggregate evidence from studies of different types and to understand the extent to which we can rely on such evidence. At present, these guidelines merely suggest that data from primary studies should be accompanied by information about the type of primary study and its quality. As yet, there is no definitive method for accumulating evidence from studies of different types. Furthermore, there is disagreement among medical researchers about how much reliance can be placed on evidence from studies other than RCTs. However, the limited number of primary studies in software engineering imply that it is critical to consider evidence from all types of primary study, including laboratory/academic experiments, and as well as evidence obtained from experts. Finally, these guidelines are intended to assist PhD students as well as larger research groups. However, many of the steps in a systematic review assume that it will be undertaken by a large group of researchers. In the case of a single research (such as PhD student), we suggest the most important steps to undertake are: • Developing a protocol. • Defining the research question. • Specifying what will be done to address the problem of a single researcher
applying inclusion/exclusion criteria and undertaking all the data extraction. • Defining the search strategy. • Defining the data to be extracted from each primary study including quality data. • Maintaining lists of included and excluded studies. • Using the data synthesis guidelines. • Using the reporting guidelines
8. References
[1] Australian National Health and Medical Research Council. How to review the evidence: systematic identification and review of the scientific literature, 2000. IBSN 186-4960329.
[2] Australian National Health and Medical Research Council. How to use the evidence: assessment and application of scientific evidence. February 2000, ISBN 0 642 43295 2.
[3] Berlin, J.A., Miles, C.G., Crigliano, M.D. Does blinding of readers affect the results of meta-analysis? Online J. Curr. Clin. Trials, 1997: Doc No 205.
[4] Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version 4.2.1. December 2003
26
[5] Cochrane Collaboration. The Cochrane Reviewers’ Handbook Glossary, Version 4.1.5, December 2003.
[6] Cohen, J. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Pychol Bull (70) 1968, pp. 213-220.
[7] Concato, John and Horowitz, Ralph. I. Beyond randomised versus observational studies. The Lancet, vol363, Issue 9422, 22 May, 2004.
[8] Feinstein, A.R., and Horowitz, R.I. Problems with the “evidence” of “evidence- based medicine”. Ann. J. Med., 1977, vol(103) pp529-535.
[9] Greenlaugh, Trisha. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analysies. BMJ, 315, 1997, pp. 672-675.
[10] Jasperson, Jon (Sean), Butler, Brian S., Carte, Traci, A., Croes, Henry J.P., Saunders, Carol, S., and Zhemg, Weijun. Review: Power and Information Technology Research: A Metatriangulation Review. MIS Quarterly, 26(4): 397- 459, December 2002.
[11] Jørgensen, Magne and Carelius, Gunnar J. An Empirical Study of Software Project Bidding, Submitted to IEEE TSE, 2004 (major revision required). http://www.simula.no/photo/desbidding16.pdf.
[12] Khan, Khalid, S., ter Riet, Gerben., Glanville, Julia., Sowden, Amanda, J. and Kleijnen, Jo. (eds) Undertaking Systematic Review of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition), NHS Centre for Reviews and Dissemination, University of York, IBSN 1 900640 20 1, March 2001.
[13] Hardman, David, K, and Ayton, Peter. Arguments for qualitative risk assessment: the StAR risk advisor. Expert Systems, Vol 14, No. 1., 1997, pp24- 36.
[14] Lawlor, Debbie A., George Davey Smith, K Richard Bruckdorfer, Devi Kundu, Shah. Ebrahim Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence? The Lancet, vol363, Issue 9422, 22 May, 2004.
[15] Pai, Madhukar., McCulloch, Michael., and Colford, Jack. Systematic Review: A Road Map Version 2.2. Systematic Reviews Group, UC Berkeley, 2002. [www.medepi.org/meta/guidelines/Berkeley_Systematic_Reviews_Road_Map_ V2.2.pdf viewed 20 June 2004].
[16] Shadish, W.R., Cook, Thomas, D. and Campbell, Donald, T. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, 2002.
[17] Jan P Vandenbroucke. When are observational studies as credible as randomised trials? The Lancet, vol363, Issue 9422, 22 May, 2004.
27
Appendix 1 Steps in a systematic review
Guidelines for systematic review in the medical domain have different view of the process steps needed in a systematic review. The Systematic Reviews Group (UC Berkely) present a very detailed process model [15], other sources present a coarser process. These process steps are summarised in Table 10, which also attempts to collate the different processes. Pai et al. [15] have specified the review process steps at a more detailed level of granularity than the other systematic review guidelines. In particular, they have made explicit the iterative nature of the start of a systematic review process. The start-up problem is not discussed in any of the other guidelines. However, it is clear that it is difficult to determine the review protocol without any idea as to the nature of the research question and vice-versa.
Table 10 Systematic review process proposed in different guidelines Systematic Reviews Group ([15])
Australian National Health and Medical Research Council ([1])
Cochrane Reviewers Handbook ([4])
CRD Guidance ([12])
Identification of the need for a review. Preparation of a proposal for a systematic review
Developing a protocol
Development of a review protocol
Define the question & develop draft protocol Identify a few relevant studies and do a pilot study; specific inclusion/exclusion criteria, test forms and refine protocol.
Question Formulation Formulating the problem
Identify appropriate databases/sources. Run searches on all relevant data bases and sources. Save all citations (titles/abstracts) in a reference manager. Document search strategy.
Locating and selecting studies for reviews
Identification of research Selection of studies
Researchers (at least 2) screen titles & abstracts. Researchers meet & resolve differences. Get full texts of all articles. Researchers do second screen. Articles remaining after second screen is the final set for inclusion
Finding Studies
Researchers extract data including quality data.
Appraisal and selection of studies
Assessment of study quality
Study quality assessment
28
Researchers meet to resolve disagreements on data Compute inter-rater reliability. Enter data into database management software
Collecting data Data extraction & monitoring progress
Import data and analyse using meta-analysis software. Pool data if appropriate. Look for heterogeneity.
Summary and synthesis of relevant studies
Analysing & presenting results
Data synthesis
Interpret & present data. Discuss generalizability of conclusions and limitations of the review. Make recommendations for practice or policy, & research.
Determining the applicability of results. Reviewing and appraising the economics literature.
Interpreting the results
The report and recommendations. Getting evidence into practice.
Get help from top-rated tutors in any subject.
Efficiently complete your homework and academic assignments by getting help from the experts at homeworkarchive.com