Assessment criteria for good scientific research, part 1

There are currently many research reports available online. These reports are prepared by students, various government agencies, and research firms. Many people assume these reports are valid and reliable. But what exactly do terms like reliability and validity mean? And what other criteria must a good research report meet? This blog post discusses these topics. Additionally, you can read part 2,, where various chapters of a good research report are discussed in detail.

Do you have any questions or additions? Please leave a comment below or contact me.

Scientific criteria for a good research report from the literature

According to Baarda & De Goede (2011) and Verhoeven (2010), an advisory report and/or scientific publication should include the following components: an abstract at the beginning, followed by a preface, table of contents, glossary, background and introduction, literature review, research plan, results, conclusion and discussion, recommendations, bibliography, and any appendices.

In the background and introduction chapter, it is essential to describe the purpose of the research and its explicit focus. Additionally, one should articulate the problem statement, research question, and any sub-questions. It is also important to outline the conceptual framework and explain the relevance or added value of the research to the scientific community. According to Kuhn (1996), anomalies often drive research. These anomalies represent phenomena that occur within a specific context but cannot be explained by the existing paradigm or framework. Ultimately, anomalies lead to paradigm shifts, resulting in the emergence of adjusted or new theories.

After the background and introduction have been established, the literature review becomes paramount. This section should elucidate the scientific framework within which the problem resides and offer insights into potential solutions based on existing studies. Theories from pedagogy, psychology, sociology, or public administration can be employed for this purpose. When selecting a scientific framework, it is crucial to consider the relevance and scientific rigor of the chosen theories (Kraemer & Perry, 1989). Furthermore, it is imperative to clearly define the theory and conduct comparative analyses with multiple theories to enhance empirical content. This approach facilitates broader generalizations and enables more robust statements (Lakatos, 1972; Maclntyre, 1985; Bailey, 1992). By generalizing, one can deepen understanding of the dynamics of social organizations and their implications for clients, employees, and policymakers, ultimately generating insights to enhance practice and address challenges (Bailey, 1992).

In the research plan chapter, it is essential to justify both the research report itself and the proposed research (Bailey, 1992; Baarda & De Goede, 2011; Verhoeven, 2010). This justification aims to ensure the replicability, generalizability, and transferability of the research, as well as to substantiate the choices made regarding data selection (Bailey, 1992; Stalling, 1986).

Generalizability refers to the ability of a study’s results to generate consistent statements about organizations or objects. Transferability means that the findings of the research can be applied to similar contexts or organizations. Replicability entails that experiments and analyses can be reproduced, ensuring the verifiability and repeatability of a study based on the information provided in the research report. If a methodology fails to meet these criteria, it is not considered scientifically robust according to Bailey (1992) and Stalling (1986).

To justify the choice of data, Baarda & De Goede (2011) and Verhoeven (2010) recommend describing the research field, specifying the objects considered for the study, detailing measurement methods and statistical techniques used, explaining the research procedure, outlining analysis methods, and describing how the data were processed.

Furthermore, it is essential to explain why specific variables were selected for measurement and how hypotheses were operationalized precisely. Contextual information, such as the location and date of the research, should be provided. Additionally, the method of verifying the outcomes of interviews and observations to ensure representativeness and substantive soundness should be clarified. When selecting variables, the principle of parsimony (Ockham, 1341) should be applied, aiming to achieve maximum explanatory power with minimal variables.

Moreover, the research plan should address the reliability and validity of the study (Baarda & De Goede, 2011; Popper, 1953). Reliability refers to the extent to which measurements are free from chance variation, while validity pertains to the degree to which the research accurately measures the intended phenomenon. In essence, validity ensures that a test or research accurately captures what it is intended to measure.

According to Lakatos (1972), internal consistency is a crucial aspect of a study, indicating the extent to which different items in a measuring instrument accurately measure the same characteristic. Additionally, it is imperative that variables exhibit correct causal relationships with each other, as emphasized by Hume (1988). Causality, as Hume (1988) defines it, involves objects being sequentially brought into relation with each other in proximity, and it’s worth noting that no two problems are exactly alike.

To enhance reliability and validity, researchers can present their findings to a panel of experts or colleagues for evaluation, as suggested by Yin (1984) and Lofland (1971). Additionally, employing multiple research methods, known as triangulation (Kraemer & Perry, 1989), can further bolster the credibility of the findings.

Moreover, Lofland (1971) and Stalling (1986) emphasize the importance of minimizing bias in research. Bias, characterized by systematic errors and issues with validity, can compromise the purity of empirical data.

The more meticulously these aspects are described, the more scientific rigor the data acquires, bringing the researcher closer to establishing scientific truth, as noted by Bailey (1992).

After detailing the research plan, the study’s results should be presented, followed by conclusions and discussions, as recommended by Baarda & De Goede (2011) and Verhoeven (2010). The conclusions should directly correlate with the results, offering a scientifically rigorous and critically reflective response to the initial inquiry. Ockham (1341) emphasized the importance of explaining study findings with minimal assumptions, discouraging ad-hoc assumptions and personal observations (Popper, 1953). Hume (1988) cautioned that scientific knowledge is inherently uncertain due to the limitations of observation and induction, underscoring the necessity of acknowledging any identified shortcomings during the research process.

Subsequently, recommendations should be provided, alongside indications of any further research that may be necessary, potentially including an implementation plan. These recommendations should not only capture specific aspects of reality but also be generalizable (Bailey, 1992). Additionally, according to Lofland (1971), maintaining internal consistency in research results is crucial, ensuring that recommendations align with the conclusions reached. It’s important to avoid normative recommendations, which dictate how things should be done (Popper, 1953).

Moreover, Kraemer & Perry (1989) assert that recommendations should be both useful and valorizing. Valorizing implies that the knowledge generated should be applicable in practical contexts. According to Kraemer & Perry (1989), knowledge is deemed useful if it addresses the priorities and needs of its intended users.

An extensive academic-level research report, of course, encompasses much more. For more detailed information on various chapters of a good research report, click here.

SOURCES

Verhoeven, N. (2010). Doing Research Like This!
Baarda, D.B.; De Goede, M.P.M. (2011). Basic Book of Methods and Techniques
Bailey, M.T. (1992). Do Physicists Use Case Studies? Thoughts on Public Administration Research
Kuhn, Th. (1996). The Structure of Scientific Revolutions. University of Chicago Press
Lakatos, I. (1972). ‘Falsification and the Methodology of Scientific Research Programmes’
Perry, J.L. (1989). Handbook of Public Administration
Popper, K. R. (1953). Conjectures and Refutations. Routledge
Wildavsky, A. (1979). “Introduction: Analysis As Art”, in: idem; Speaking Truth to Power
Humme, H. (1988). The Nature of Man (Hebrew Edition)
Ockham, W. (1341). Ockham’s Summa Logicae
Yin, R.K. (1984). Case Study Research: Design and Methods. Sage, Newbury Park
Lofland, J. (1971). Analyzing Social Settings