1/3/2008

1. Methods and Techniques for developing Research projects

A possible vision about Theories on the development and
characterization of Scientific Research

Eugénio Oliveira- 2007/2008

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

1. Abstract of my Lectures

Understanding about Science

Scientific Methods and Techniques
Writing scientific papers
Evaluating scientific papers
Consulting scientific Data Bases and searching for information ( Dr. Ana Azevedo)

2. Abstract of your Work

Criticizing a scientific paper

Organizing a workshop
Writing a scientific paper
Presenting a scientific paper

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

1
 1/3/2008

1.1 Basic Motivations: Issues on Scientific

Methods

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Source: GETA and Graduate School of Electrical and Communications Engineering

course: Introduction to Research Methodology, Aarne Mammela, VTT-
Helsinki University of Technology,

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

2
 1/3/2008

• Basic Characteristics:
• Cumulative versus reformulating always from the principles

• However:
•Thomas Kuhn
(Harvard, Berkley, Princeton, MIT until 91. +96)
“The Structure of Scientific Revolutions” 1962.

• He points out the role in scientific research of the

emergence of NEW PARADIGMS.

•A PARADIGM is defined as:

Scientific discoveries, which universally
recognized during a certain period of time, create
a new model for both problems and their
respective solutions to be used by a community
of practicians ans experimentalists
Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Paradigms support theories based on concepts, phenomena and

techniques for helping on explaining new facts or information
.

• Science progresses through successive ruptures depending on

the current accepted paradigms.

Paradigms correspond to “different views” over the reality which

are influenced by aspects other than scientific ones.

Wittgenstein and duck-??????…

L.Wittgenstein “Tractatus Logico-Philosophicus

Wttgeinstein’s Poker”, David Edmonds, J. Eidinow

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

3
 1/3/2008

• Scientific interpretation (and perception) is mostly affected by

Social as well as Psychological factors.

• Does it lead to Irrationality dominating Science?

No !
• A good discussion on prevalence of science can be found in:
• Kuhn, T.S.
“Black-Body Theory and the Quantum Discontinuity”, 1987.

However: Paradigm change non-cumulative Science (sometimes)

• Example of important Paradigm changes:

• Solar System Ptolemy's vision changed by Copernicus

vision
• Going from Newtonian Phisics to Quantic Physics
(Uncertainty Versus Determinism..) and Relativity (Space-
Time relationship).
Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Karl Popper (London S. E., + 94)

• “The Logic of Scientific Discover”, Basic Books, 1959 (1ªEd Ing)

• Conjecture and Refutations, Harper & Row, 1968

• Idealistic. He developed a Scientific Method as an Epistomology

(a way of “knowing”) independently of the intrinsic value.

• T. Khun e Feyerabend (1993) criticize this point of view showing how

science is guided by value and, therefore being non-idealistic.

• This is a debate on “Consciousness over Existence”

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

4
 1/3/2008

• “Critical Rationalism” proposal defends “falsifiability”

as a criterion for demarcating (separating) science and non-science

Induction Baaaad!!! - Falsifiability gooood!!!

• Popper argues that scientific theory will always be conjectural and
provisional.

It is not possible to confirm veracity (the truth) of a theory by means

of simply recognizing that the results of a preview already done based
on that very same theory, hold.

That Theory should only be seen as a theory not (or not yet)
contradicted by facts.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Verificationism:
a sentence should be either verifiable or falsifiable
Critics addressed to Verificationism:
Generalization: specific instances lead to universal
assumptions

Observation X, being an instance of Theory T

Increases the probability of Theory T being correct
The “Black Crow paradox”:
• The “Black Crow paradox”: by Hempel:
“All Crows are Black" is logically equivalent to
“Everything which is Non-Black is not a Crow".
∀x(Cx Bx) ∀ x(¬Bx ¬Cx)
• Thus, If ∃ x(Cx ∧ Bx) reassures that ∀ x(Cx Bx),
Then ∃ x(¬Bx ∧ ¬Cx) reassures that ∀ x(¬Bx ¬Cx)
which, being equivalent to ∀ x(Cx Bx),
this last implication would also see its probability
of being True increased.
Eugénio Oliveira- 2007 MSR-Metodologias
MIC- Methodologies
defor
Investigação
Scientific Research
Científica

5
 1/3/2008

• Crítics addressed to Verificationism:

• Generalization: specific instances lead to
universal assumptions
• “There are Apples (non-Crows) not Black”
Increases the probability that the sentence
“All Crows are Black” is True

• If verificationism (generalization) was

acceptable, any trivial evidence would support
a Theory
There are attempts to overcome this paradox..
“falsifiability” proposed by KP states that keeping
just the Existential assertion permit us to know
that if another existential proposition that contradicts it, arrises,
this falsifies the Theory T !!

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

6
 1/3/2008

A
An exploration Enterprise:

Christopher Columbus – the “Explorer”

Motivation: ambiguous…
Problem: a new way to “India”
Hypothesis: over the Atlantic and not around
south Africa
Experiment: Yes he have done it !!
Conclusions: ?????

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

A
research: discover new knowledge

• basic research (no specific application in mind)

• applied research (ideas into operational form)

• development: systematic use of the existing

knowledge

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

7
 1/3/2008

A
Science, Technology and Engineering [Jain97]

• science: organized or systematic body of knowledge

• technology: application of scientific knowledge for

practical
• ends in engineering, medicine, agriculture, etc.

•natural sciences and engineering sciences differ in

the object of study:

• natural sciences: objects in the nature

• engineering sciences: objects (products,
services, methods) not found in the nature, using
results of mathematics and natural sciences
Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

A
Science, Technology and Engineering [Jain97]

Verification, confirmation, and justification are

synonymous terms in philosophy of science. The
opposite is falsification or refutation.
R. K. Jain and H. C. Triandis, Management of Research and Development
Organizations: Managing the Unmanageable. John Wiley & Sons, 1997.

Requirements for success

• You have to be:
• analytical and curious,
• autonomous and collaborative
• criticism and tolerant

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

8
 1/3/2008

A
How does a researcher work?

• make always notes in a “notebook”

• • make summaries on what has been learned.
• make plans for the future all the time (outlines,
roadmaps)
• discuss, ask questions and argue (criticism)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

A
Learning carefully by induction (bottom up, generalization
from examples to models) [Felder88]

• Defending theories by deduction (top down, from

models to results)
•
Break the problem down and then generalize the results
(divide and conquer)

R. M. Felder and L. K. Silverman, “Learning and teaching styles in

engineering education, ”Engineering Education, pp. 674681, April 1988.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

9
 1/3/2008

1.2 Supporting Theories

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Basic Characteristics

• Research involves Methodologie(s) and Techniques

• A- Methodology includes the study of several methods that are

applicable to a class of problems
• i.e., the set of processes to conduct each specific Research Project

• B- Techniques refer to the means and specific Tools

enabling relevant information acquisition, the respective data
analysis, as well as the inferences that can be made.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

10
 1/3/2008

• A- Methodologies
• depending on the MOTIVATION:

• ‘Pure Research’:
• It contributes to a deep theoretical understanding and for a
more abstract formulation of the phenomena.
• ‘Instrumentalist Research’:
• It contributes for making human intervention in Real world
environments, more effective.

• Two major sub-categories are identified:

• ‘Applied Research’
• ‘Problem - Oriented Research’

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• ‘Apllied Research’.
• Starting from a technology (devices, specific techniques, both)
use them for dealing with processes (physic, organizational, social,
individual,...)

• ‘Problem- Oriented Research’.

• Starting with the problem description and then looking for
the adequated techniques to solve it.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

11
 1/3/2008

• Depending on the Theory underlying the Research:

• ‘Descriptive’, of all significative aspects of the domain

• ‘Explicative’, of the behaviour of all the phenomena

• ‘Predictive’ of the future phenomena behaviour

• ‘Prescriptive’, beyond the prediction it should also prescribe and apply

Norms and processes in identified specific circumstances

(Research in IS, Electronics and Telecommunications should be of

this kind)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Depending on the tradition in that scientific area or Knowledge Domain:

i) ‘Traditional Scientific Research’ which is more quantitative

ii) ‘Interpretative Research’ which is more qualitative

iii) ‘Engineering-oriented Research’

• iii) most appropriated for Electronics, Telecommunications and CS

• But do not ignore the other approaches!

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

12
 1/3/2008

1.3 Positivist (classical) Research method

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Traditional Scientific Research is based on positivist rational

thinking:

• From observations we may build up Theories trying to

explain what has been observed

• Theories are expressed in Deductive form or using

Axioms and Postulates which are then operated through
Logics.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

13
 1/3/2008

Do Not Forget :

• Scientific Theories, following K. Popper, are capable of

generating Inferences which are, in principle, falsifiable
through references to the real world.

• Results coming from the process of testing hypothesis

give feedback to the Theory, enabling verification of its
trueness or detecting possible “ab-normalities” (T.Kuhn).

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Whenever the Observation comes from the Real World it is called empirical
and it becomes relevant in two situations:

• during the formation of the theoretical knowledge through the so-called

passive observations

• during Hypothesis verification, gathering information in a more

active way, although guided by that same theory.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

14
 1/3/2008

• When we do not have a suitable Theory available, we have to pursue

‘exploratory research’ where experiments and interpretation are open,
not guided by a Theory and making available Empirical Knowledge
that may postulate other Theories.

• IS and Telecommunications are disciplines (scientific areas) without a

given stable theory.
• They encompass applied subjects using a partial theory or co-opting a
theory from “reference disciplines” such as:

• Mathematics
• Information Theory (C.Shannon)
• Physics
• “Theory of Organizations”,
• “Management Sciences”,
• “Computer Science”.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

3
Theory of
Disciplines
of Reference

2 Theory
Axioms
Postulates 4 7
Deductive Logic
Simulation

Research 6
Project
Hypothesis

5
1 Real World

1- There exists a Real World that we can Perceive.

2- Knowledge is Organized in Theories making inferences possible
3- Applied Disciplines like IS or T rely on Theories of Reference of other Disciplines
4- A Research Project permits testing formal Hypothesis formulated through inferences expressed in the
Theory.
5- In Empirical Research, Hypothesis derived from the Theory are directly cheked in the Real Worldl
6- Testing Hypothesis through Simulation
7- Results comming out of the Research give feedback to the Theory

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

15
 1/3/2008

• Summarizing: In traditional (or conventional) Science, new Hypothesis are

extracted from the existent Theory, they are tested and new results are
added to that Theory.

• Thus, it implies the pre-existance of:

• A theoretical Body of Knowledge
• An explicit theoretical Framework to guide the research
• Definition of what issues have to be investigated
• What are the explicit Hypotheses that could be refuted (falsifiability)
• Method for Applied Research and well defined techniques
for testing the Hypothesis.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

1.4 Interpretative Research method

is concerned with trying to understand lived

experience and with how participants themselves
make sense of their experiences. Therefore it is
concerned with the meanings which those
experiences hold for the participants. IRM is
phenomenological in that it wishes to explore an
individual’s personal perception or account of
an event or state as opposed to attempting to
produce an objective record of the event or state
itself.
http://www.psyc.bbk.ac.uk/ipa/ (Birkback- U.
London)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

16
 1/3/2008

• Interpretative Method (or philosophy):

• post-positivist
• criticizes the (conventional) scientific “chimera”
• declares the impossibility of objective observation
• Observations and Interpretations depend on the Observer.
• Addresses critics to both “hard Sciences” and Social Sciences,

• Difficulties for Objectivity:

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

• Concepts, Factors, relationships that can’t be accessed

• Researchers involvement in the research domain

• Results depending on the researcher own perspective:

- On selecting and defining the research (sub-) domain
- On selecting a pre-existant theory
- On setting the issues to be investigated
- On setting the precise scope of the research
- On selecting and defining the Variables to be instantiated
- On measuring those variables possible values

• Different attitudes on all those issues may lead to many different

interpretations of the same observed phenomena.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

17
 1/3/2008

Engineering-oriented Research Approach

1.5 Research in Engineering disciplines

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Engineering-oriented Research Approach

• For this approach to research, technology is very important!

• (devices, artefacts, practical techniques to be used)

• At least up to a certain extent is the most suitable for research in

Information Systems, Electronics, Telecommunications.

• Research in IS and Telecomm is applied and problem-oriented.

• Implying:
- application of suitable technology,
- conceptualization: requirements, models
- prototyping,
- implementation
- demonstration of (new) technology (real world/sim)
- Evaluation phase

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

18
 1/3/2008

Engineering-oriented Research Approach

Observation and Theory [Wohlin99]

• In engineering an hypothesis (defined in system specifications)

is usually an idea of the relationship between the cause and effect
(defined in system requirements)

• Theoretical model is always only an approximation of observation

in real world

C. Wohlin et al., Experimentation in Software Engineering: An Introduction, Springer, 1999.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Engineering-oriented Research Approach

In engineering we are fundamentally interested in

how efficiently the basic resources are used for a
needed performance.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

19
 1/3/2008

Engineering-oriented Research Approach

• From Dr. Leung Yee Hong Australia TC Research Institut

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

1.6 Techniques associated to Research

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

20
 1/3/2008

Research Techniques

B- Techniques

I. Non empírical Techniques

II. Scientific Research positivist Techniques

III. Scientific Research Interpretative Techniques

IV. Scientific Research Techniques at the positivist/interpretative boundery

V. Scientific Research Engineering-based Techniques

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

I. Non empirical Techniques depend on:

• Artificially created data, or

• Conceptual thinking about abstractions

• Includes:
• conceptual research based on opinion and speculation.
( argumentative and dialectic analysis)
• theorem proof: This applies to mathematical abstractions:
• formal methods application,
• induction,
• mathematical abstractions
• Models verification
• Simulation:
• Mapping a complex environment in a simplified Model.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

21
 1/3/2008

Research Techniques

• Other non empiric Techniques:

• Using hypothetic future scenarios

• Mapping a real scenario into Games
• ...

• Reviewing the State-of-the-art and doing Meta-analysis

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Março 2007 FEUP-LIACC / Eugénio Oliveira

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

22
 1/3/2008

Research Techniques

II. Scientific Research positivist Techniques

(applicable to IS and other Engineering disciplines):

• Techniques for forecasting applying

• regression algorithms and
• time-series based algorithms to extrapolate on historic
data
• Field Experiments :
• Possibly isolating and controlling a set of phenomena
to be looked at

• Laboratory Experiments:
• Creating artificial setups in which phenomena, factors
and variables are isolated and controlled.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

III. Scientific Research Interpretativist Techniques:

• Research approach is Descriptive/Interpretative:

Going from empirical observations to a limited rigour analysis.
Need for controlling the researcher’s perspective:
- Critical self- Examination
- Revising all the postulates and biases
- Varying Observations
- Submission to peer reviewing

• Group Research:
• frequent discussion with a group of people including
who may be affected by the application (or the technology)
• Use Collaborative Work Tools.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

23
 1/3/2008

Research Techniques

• Action-based Research:

- Researcher is like an “Agent” changing existent

conditions to measure the respective reaction

• Other Methods for Social Sciences(Ethnographic..)

In http://www.qual.auckland.ac.nz/ Michael Myers Ed. Ass.Inf.Sys.

It is advocated qualitative research, for example, for IS.

We also can do (but it is not recommended) R&D just through

Case-study analysis

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

IV. Scientific Research Techniques at the positivist/interpretativist

boundery

Including:
- Field Work : Data related to the object of study is
directly observed by the Researcher in the original
context
- Questionary-based : Structured Data collection comming
from interviewees

- Case Study: - Isolate and describe a relevant real-world

situation,
- Gather a collection of data from multiple
sources

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

24
 1/3/2008

Research Techniques

• Secondary Research:

• Analysis starting from pre-existent documents (texts,

papers, descriptions, memoirs).

• Analysis is done under new perspective and using

different techniques or procedures.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

v. Scientific Research Engineering-based Techniques

(Informatics Engineering, IS, Electronic Engineering, Telecomm...)
• Divided into two categories:

1- Construction Technique:
• Includes Design, Project, Implementation
(or prototyping) of a system
(ex: Computer program System, or physical device)
- objective:
-- explicitly testing an hypothesis or
-- to solve a class of problems.
- it implies:
-- Existence of either a theory or a model
for explaining the results
(testing the hypothesis)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

25
 1/3/2008

Research Techniques

Materials and Methods

• in exact sciences materials include definitions,

model, and related assumptions

• theoretical model is defined (preferably a mathematical model),

including boundary conditions (define the environment)
and initial conditions (define the initial values of the
parameters of the model) [Losee01]

J. Losee, A Historical Introduction to the Philosophy of Science,

4th ed. Oxford Univ. Press, 2001.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques

• methods include:
rules of analysis and
rules of verification and validation

• rules of analysis follow deductive reasoning,

statistical analysis and approximations (analysis
must often be replaced by simulations due to
mathematical tractability problems)

• the theoretical model is verified by comparing the results

with reality (measurements with a prototype, known as
hypothetical-deductive method)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

26
 1/3/2008

Research Techniques

• Results

• results can be analytical (deductive), from simulation, or

measurement results

• it is important to present numerical results that

verify (or not) your own thesis,

• use analytical results in simple limited cases to obtain reference

values

• the value of the results depend on how well they can be

generalized (bottom up approach will help you)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research Techniques
Conclusions (constructive technique report)

(1) “We have shown that [some] algorithms proved right…”

(2) “Our approach is simpler to implement than previous algorithms and is

practically feasible …. On the theoretical side, the greatest deficiency in our
results is …”

(3) “In a practical situation the advantages and limitations of our

approach are…”

(4) “These results can be extended in a number of directions…”

1. What is shown by this work and its significance.

2. Limitations and advantages.
3. Applications of the results.
4. Recommendations for further work.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

27
 1/3/2008

Research Techniques

• Destructive Techniques:

• To analyse a system, or a set of methods to gather

new information about those technology procedures
under analysis.

- Recognized information and characteristics are

then reused may be in a different way

- Example: Reverse Engineering

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Phylosophy Underlying Research

• There are philosophies permitting the Classification of Research

Methods through Paradigms or basic Epistemologies:

I. Positivist
II. Interpretative (post-positivist)
III. Critical
IV. (Constructivism)

At Harvard Business School they use just the three categories

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

28
 1/3/2008

Philosophy Underlying Research

• Positivist Method:

• Assumes that reality is objectively accessible, it can be

measured and described by the observer and measured by
the same observer through their instruments and devices.
Tests are made making predictions possible over future
phenomena
• IS research (also Telecomm research?) can be of the
positivist kind (for IS, Orlikowski and Baroudi,1995)
- Use math/logic formalisms
- use quantifiable and measurable variables
- Test an Hypothesis and make inferences from what is
known.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Philosophy Underlying Research

• Interpretativist Method

• Assumes that accessing to reality is only possible through

social constructions like language, consciousness and
ontologies (sharing of concepts’ meaning).

• Hermeneutics-based philosophy (based on the human

understanding and text interpretation) as well as in
phenomenology
(describing, understanding and interpretation perceived
phenomena)

Proposes to abolish separation between "subject“ and

"object",
It opposes to XIX century positivist thinking.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

29
 1/3/2008

Philosophy Underlying Research

• Can IS and Telecomm studies influence or be influenced by

the context?

• Here, the focus is not as much on the variables,

measurements and processes as it is on interpretation of
what is being the results of the test, the measurement or
implementation

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Philosophy Underlying Research

• Research Criticism:

• It is a radical enterprise once it tries to identify what constraints

research:
• socially, culturally,
• Historically, politically,
• economically, physically
trying to emancipate science by eliminating causes for ignorance,
alienation etc...
Ex: Why is it not possible to do research on X?
Ex; Why cannot we implement systems of type Y?

Habermas from Frankfurt School following the way of Adorno,

Lukacs, Marcuse

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

30
 1/3/2008

Philosophy Underlying Research

• Junk Science (?) Stories of the Past Decade

Dial “F” for Fear.

Since the 1993 Larry King Live broadcast featuring a man suing a
cell phone maker claiming his wife died from a cell phone-
induced brain cancer, many cell phone users have worried
about phone safety.

But studies failed to identify any risk.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Philosophy Underlying Research

• Junk Science (?) Stories of the Past Decade

Powerline scare unplugged.

Fears that electric and magnetic fields (EMFs) created by power

lines and appliances caused cancer started in 1978. Parents
worried about power lines over schools.

Power companies worried about burying power lines. The National

Academy of Sciences finally unplugged the scare in October 1996,
concluding that no evidence showed EMFs presented a health
hazard

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

31
 1/3/2008

Philosophy Underlying Research

• Junk Science Stories of the Past Decade

The Lone-Tree Theory.

It nearly took an act of Congress to get the researcher behind the

notorious “hockey stick” graph, which purports to show a steep
rise in global temperature in the 20th century following a
millennium of stable temperatures, to release his publicly funded
data and computer code.

Among other dubious presumptions, the graph is derived from

data that bases climate estimates for the entire 15th century on
the tree ring measurements of a single tree.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Quality of Research

1.7 Quality of Research

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

32
 1/3/2008

Quality of Research

• Quality depends on the Objectives to reach

1. To increase available (and also general) Knowledge
• Objectives and Quality criteria are internal to the discipline
• RIGOUR is very critical

2. Developing Computer Systems– (Instrumentalisme)

• Objectives and Criteria are external (regarding R&D)
• Relevance can be measured by the impact and is of the
most importance

• Relevance Versus Rigour

• Not antagonic but taken with different emphasis

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Quality of Research

• Computer Science (and other Engineering-like areas) tendency:

• Until the 80ies conceptual (not empirical) papers

dominated
• After the 80 ies Less theoretical papers, more
instantiated, less general and Knowledge less cumulative

• Case-studies based research is mainly descriptive concerning

simple instances and then less scientific

• Empirical research is now dominated by the descriptive

approaches

• Specific Models have limited capabilities for explaining a

phenomen as well as little predictive power.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

33
 1/3/2008

Quality of Research

• Requirements for the Research in CS and TC

• Choose an appropriate Research Method:

• Traditional/positivist, “interpretativiste” or
Engineering-oriented

• Make explicit a Body of Theory sometimes coming from

other adjacent disciplines (ex: Computer Science, Organization
Theory, Information Theory, Physics, Statistics…)

• Extend the Theory and Infer predictions to be checked

(at least in the case of positivist/traditional research)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Scientific Methods, General

(following Björn Lisper, Datorteknik, Mälardalens högskola

Västerås, Sweeden)
http://www.idt.mdh.se/kurser/ct3340/ht03/

Theoretical methods:

Create formal models (mathematics, logic)

Define concepts within these

Prove properties of the concepts

Abstraction, hide details to make the whole more understandable

(and to make it possible to prove properties of it)

Proofs of properties by deductive methods

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

34
 1/3/2008

Scientific Methods, General

Empirical methods:

Perform experiments

See how it turned out

Draw conclusions

Simulation:
Start with a formal model at some "easy-to-understand" level

Make "artificial experiments" in your computer

Collect statistics and draw conclusions

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Scientific Methods

In Physics:
Make hypotheses about the surrounding world (theory),
observe it (experiment)
Relate the result of experiment to the theory

Adjust the theory if it doesn’t predict the reality well enough

Theory is used to predict the future (e.g., if a bridge will hold for a
certain load, or an asteroid fall down on our heads)

Common pattern in Computing Science:

The system is constructed to behave according to some
theoretical model
Deviations are seen as construction errors rather than
deficiencies in the theory (hardware error, bug in OS, : : :)

In both cases: the theory helps us understand and predict, but in

different ways!

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

35
 1/3/2008

Scientific Methods
Theoretical vs. Empirical Methods in Computing Science

Computing Science really has a “spectrum”, from “extreme

constructivism” to a use of theory close to the one in physics:

• “Extreme constructivism”: (ideal) programming language design:

Formal semantics for the language, pure construction of model

defining the mathematical meaning of each program

Abstraction of details to make the meaning of the language

simpler (for instance, assume that data structures can grow
arbitrarily big)
Implement the language according to the semantics

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Scientific Methods
Theoretical vs. Empirical Methods in Computing Science

One can prove formally within the model that a

program is correct –valuable!

But the model does not cover all kinds of failures.

E.g., hardware errors, or stack overflow (or an
asteroid falling down on the computer)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

36
 1/3/2008

Scientific Methods
Theoretical vs. Empirical Methods in Computing Science

Extreme “physics” approach:

performance modeling of complex computer-
and communication systems

Extremely hard to make analytical calculations

Simplified performance models, tested against
experiments (e.g., long suites of benchmarks)

Discrepancy leads to a modified theory, as in physics

Often simulation (desire to evaluate systems before
building them)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Scientific Methods
Theoretical vs. Empirical Methods in Computing Science

In-between: algorithm analysis

Build on some form of formal model for how the

algorithm executs (meta-language with formal
semantics), and some performance model (how long
does a step in the algorithm take, how much memory
is needed to store an entity)

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

37
 1/3/2008

Scientific Methods
Theoretical vs. Empirical Methods in Computing Science

In-between: algorithm analysis

Given that the performance model is correct, one proves

mathematically that the algorithm needs certain
resources (time, memory) to be carried out
But the performance model is often very approximate.

Sometimes is possible to refine the performance model,

but this can make it impossible to calculate the resource
needs of the algorithm

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Quality of Research

• Characteristics of the Research in CS and TC(?)

• Report the S-o-A.

• Prove you made an advancement in Knowledge
• Combination of Techniques trying to compensate one’s
weaknesses with other one strengths
• Report about the object of study and the nature of the domain
• Validate Data
• Analysis using statistics
• Extract relationships between variables

• Praticable according to the resources

• Relevant results (at different levels: community,society...)
• Publish
• Ambitious

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

38
 1/3/2008

Quality of Research

• Characterístics of the research in CS:

• Research may have as an objective:

• Building a Theory
• Testing a Theory
• Extending a Theory

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Quality of Research

• Other characteristics:

• Concerning Data:
• Extensive
• Representative
• What is Data validity (ex. temporal, precision…)
• Granularity:
• Simple or agregate
• Specific or large generic experiments

• Temporal Horizon:

• “Snapshot” of the domain at a specific point in time

• “Longitudinal cut section” (múltiple points in time)

• Methods for colecting and analising data; quantification...

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

39
 1/3/2008

Quality of Research

1.8 Challenges

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research in CS and TC Challenges

• Challenges:
• There exists shortage of theory. Mainly theories in related disciplines
• Phenomena are unstable:
• Data may get old quickly, usually a snapshot that may vary
Pay attention to the validity period
• Organizations and Markets are always moving

• Significant cultural variations:

• Different national/regional cultures imply different speed/rhythm

• When you are doing research (requirements, interaction,

questionnaires, demos) you influence the domain

• Pressure for publishing during the research action

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

40
 1/3/2008

Research in CS and TC Challenges

• Pressure for technical instantiation:

• Prototyping without a theoretical ground or well defined

objective
• Economics and business models either not known or consolidate.

• Impact of the automation and rationalization:

• Depends on time dynamics concerns
• Depends on who are directly affected

• Adopting a new technology may take long time

• Impact on other partners, markets, standards, tools…

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

Research in CS and TC Challenges

• Conclusions on Research actions in CS and TC

• Consider tradoff Rigour Versus Relevance

• Identify challenges BEFORE starting doing research
• Be clear about conventions related with research
to be done
• Select a significative sub-domain
• Define questions to be answered through that research
• Select the apropriate Research Method and Techniques
(possibly a good combination of some) and justify the choice.

Whenever the selected aproach is instrumentalist, you need

to be sure about the external quality, data validation and rigour
as well as:
Relevance

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

41
 1/3/2008

Other Bibliography

• D. Sternberg, How to Complete and Survive a Doctoral Thesis.

St. Martin’s Press, 1981.

Eugénio Oliveira- 2007 MSR- Methodologies for Scientific Research

42