Project Topics
Get Complete Project Material File(s) Now! »
Research Methodology
A (SLR) systematic literature review [17] is a well-defined approach to iden-tify, evaluation and interpreting all relevant studies regarding a particular research question, topic area or phenomenon of interest. Usually, systematic reviews conduction is a three-step approach. Figure 2.1 shows the main steps composing the SLR process regarding the Planning, Execution, and Result Analysis [22].
Two researchers are involved in this systematic literature review. In or-der to ensure a consistent planning process of our investigating systematic literature reviews and to reduce the possibility of researcher bias, we have developed a study review protocol.The main purpose of this protocol is to provide a common structure for our study review. In the planning Phase , the review protocol is defined, which includes the definition of research ques-tions, the search strategy and scope, the data items that had to be collected, the approach for data analysis and presentation of the results. The research questions express the research topics of interest in this literature review. The scope of the review was based on the identification of the main workshops, conferences, and journals in the field. Also, we proposed our strategy for accessing the quality of studies that we considered in the review. Next, data items were identified and for each item a set of options were defined. The definition of data items was based on information derived from liter-ature sources and from experiences with a preceding literature review [32]. For some of the data items, additional options were introduced during the review process, in particular for the fields of software engineering and vali-dation setting. The protocol was cross -checked by an external reviewer and the feedback was used to make small improvements. Finally, the data de-rived from the primary studies was collected and summarized to answer the research questions.
The final report was checked by one independent researcher. His feedback was used to improve the description and correct minor issues.
Research Questions
We aimed at research questions meaningful not only to researcher but also to engineers. Thus, we first formulated the review goal through Goal-Question-Metric (GQM) perspectives (purpose, issue, object, viewpoint) [4]:
Purpose: Analyse and characterize.
Issue: Claims and Evidence.
Object: for self-adaptive software systems.
Viewpoint: from a researcher’s and engineer’s viewpoint.
The general research question translates to three concrete research questions:
RQ1: What are the claims made for self-adaptation?
RQ2: What are the tradeoffs implied by self-adaptation?
RQ3: How much evidence is available for the claims and what are the types of evidence?
The goal of RQ1 is motivated by the need to get clear understanding of the claimed benefits of self adaptation and current state-of-the-art in self-adaptive systems. We are interested in identifying that how many quality attributes addressed in self adaptive systems and the domains of application in which self-adaptation has been applied. RQ1 is important for researchers and engineers to get an overview on quality attributes of the self-adaptive systems. RQ2 is to understand about what are the tradeoffs by applying self-adaptation.We investigate in this question which quality attributes have positive or negative influenced on self-adaptation, and whether or not the researchers clearly defined tradeoffs of software quality. With RQ3 we aim to investigate what assessment method have been used for evidence and how much evidence is available for applied methods. By evidence we mean ev-idence for adoption of methods of researcher that they used to prove their claims. In this research question we also investigate what assessment ap-proaches can be useful for increasing the level of evidence. We also assessed the quality of the study. From the researcher and engineers point of view RQ3 give us an overview of evidence and what purposed method could be useful in practice.
Search Strategy
Our search strategy combines automatic with manual search. Our search comprised of three steps: In a first step we searched the studies by automatic method and in 2nd step manually we selected the studies that are relevant for self-adaptive systems, and then we filtered the engineering studies that have any evidence. We used the following search string in the first step:
(( Title:adaptive OR Title:adaptation OR Title:self OR Title:autonomic OR Title:autonomous ) OR ( Abstract:adaptive OR Abstract:adaptation OR Abstract:self OR Ab-stract:autonomic OR Abstract:autonomous )).
To ensure the validity of the search string we applied pilot searches on the set of studies from three specialized venues TAAS, ICAC, and SEAMS to ensure that the keywords provide the right scope of studies.
One researcher is responsible for the automatic search. Automated search performed by executing search string on five electronic data search engines, these are: IEEE Explore, ACM Digital library, Science Direct, Think Mind(TM) Digital library. Manual search refers to performed manually browsing Con-ference proceedings or Journals proceedings. All search were based on title and abstract.
We further refined the studies resulting from automatic search using a manual search step. The goal of this step is to identify the primary studies that are directly related to the research questions.Manual search was per-formed independently by both reviewers and disagreements were resolved by discussion. To that end, we defined the following inclusion/exclusion criteria.
Inclusion Criteria
We limited our literature search over two dimensions: time (publication pe-riod) and space (publication venues).
Studies which were published between 1st of January 2000 to present. We decided this broader period due to the reason that self-adaptive systems have become subject of active research around that time.
The study must be related to the field of self-adaptive systems that must addressed adaptation logic in which after engineering self-adaptive software systems performs adaptations of the managed system when needed.
We included only those studies that provide any evidence and assess-ment method to prove their claims. Assessment are validation ap-proaches that may be in the form of example application, simulation, rigorous analysis, empirical, real world example or studies that provided some qualitative, textual, opinion-oriented evaluation. E.g. compare and contrast, oral discussion of advantages and disadvantages. Evi-dence may be in the form of toy example, observations, experiments, empirical or industrial evidence in Example application Authors de-scribing an application and provide an example to assist in the descrip-tion, but the example is « used to validate » or « evaluate » as far as the authors suggest [27].
Exclusion Criteria
We excluded those studies which are related to theoretical such as new theory about self-adaptive system, discussion on new algorithm and do not provide any evidence about their theory because, we are only interested in those studies that have certain level of evidence.
We also excluded tutorials, short papers, editorials etc. because these papers do not provide a reasonable information.
A study selected when it met the inclusion criterion and eliminated if it met any of the exclusion criterion.
Sources searched
To ensure a minimum level of quality of studies, we include the primary venues for publishing research results on self-adaptive systems, as well as the major conferences and journals on software engineering. The selected venues are listed in Table 1. The Rank is based on the evaluation published by the http://www.arc.gov.au/era/era_2010/archive/era_journal_list.htm. Australian Research Council. And those venues are not ranked by Australian Research Council, we put there n/a which means not applicable ranked.
Study Selection Process
The main purpose of study selection process was to identify the relevant studies. The search string was used on the search engines to find the related studies from the selected sources in Table 1. Filtering of the studies selection process on each stage are described fully in figure 2.2.
At stage 0 all the venues were automatically searched by search string and downloaded all of the retrieved studies. After that at stage 1 both of re-searchers collaboratively manually checked the title , keywords and abstract of all the retrieved studies, discarding if any study did not meet the selec-tion criteria. It may be possible that at stage 1 we also added all of those studies that were doubtful weather is included or excluded. Brereton et al. argue that abstracts might be too poor to rely on when selecting studies [5]. Thus at stage 2 we also decided about study inclusion based on the conclu-sions of studies and also as well as review the introduction, if needed. Then we obtained the full printing copies of all remaining studies. Both of the researchers individually taken the data extraction of each study and then both compare their results, a decision for each study was taken based on consensus.
By reading the details of the full text studies, more studies excluded, which were not relevant according to inclusion/exclusion criteria. Finally remaining studies were selected for data extraction that used in this study.
Quality assessment
Assessing the quality of the paper or its contribution is important for data synthesis and interpretation of results later on. As all studies were assessed through a quality check, To assess the quality, we collected a set of quality items as show on Table 2. These items are based on the assess method for research studies proposed in [12]. From the answers, a quality assessment score (max 12) is calculated by summing up the scores for all the questions for a study (scores for the various options are given between brackets).
Table of contents :
1 Introduction
1.1 Problem context
1.2 Self adaptive system
1.3 Contribution and Objectives
1.4 Related Work
1.5 Report Structure
2 Review Protocol
2.1 Research Methodology
2.2 Research Questions
2.3 Search Strategy
2.4 Inclusion Criteria
2.5 Exclusion Criteria
2.6 Sources searched
2.7 Study Selection Process
2.8 Quality assessment
3 Data Items
3.1 F7. Subject of the study
3.2 F8. Application Domains
3.3 F9. Quality Concerns
3.4 F10. Claimed benefits of self-adaptation
3.5 F11. Tradeoffs
3.6 F12. Assessment / validation approach
3.7 F13. Evidence level
3.8 F14. Repeatability
3.9 F15. Validation setting
4 Results Analysis
4.1 Included studies
4.2 RQ1: What are the claims made for self-adaptation?
4.2.1 Classification of studies according to the software engineering fields and years
4.2.2 Statistical analysis of application domains.
4.2.3 Classification of studies according to the quality concerns and year
4.2.4 Comparison of correlation between application domain and quality concerns.
4.2.5 Summary of learned lessons from RQ1:
4.3 RQ2: What are the tradeoffs implied by self-adaptation?
4.3.1 Summary of learned lessons from RQ2:
4.4 RQ3: How much evidence is available for the claims and what are the types of evidence?
4.4.1 Classification of studies according to the assessment methods and year
4.4.2 Statistical analysis between assessment methods and number of primary studies on each year.
4.4.3 Statistical analysis of repeatability of study.
5 Discussion
5.1 Limitations of study
5.2 Personal reflections on the review
5.3 Conclusion
5.4 Future Work
References
