Transcript Slide 1

Week 5. Research Questions, Goals,
Experiments and Methodology
MSc Methodology Seminar I
Dr. Felipe Orihuela-Espina
Outline
1. What is a thesis?
2. How to elaborate an MSc thesis
1. How to choose a topic
2. The actors
3. How to write the thesis
3. Main parts of a thesis
1. Research Questions
2. Goals
3. Experiments and Methodology
17/07/2015
INAOE
2
WHAT IS A THESIS?
17/07/2015
INAOE
3
What is a thesis?
 A thesis is:
 Original (creative) and significant
 A record of research (so that it is reproducible)
 A critical survey
 Reviews existing work
 Discusses results in terms of current evidence
 The conclusion of a set of coherent experiments
for testing certain hypothesis about a certain
phenomenon of interest.
17/07/2015
INAOE
4
What is a thesis?
 A good thesis includes:
 Who? – A phenomenon being studied
 What? – A clear message, hypothesis and/or
claim
 …and evidence for this
 Why? – A motivation and justification
 How? - Methodology
 When (time) and where (spatial)? – Constraints,
limitations and discussion
17/07/2015
INAOE
5
Common pitfalls while doing a thesis
 Being too ambitiuos
 Stopping reading after an initial survey of
related work
 Think that there is no related work
 You simply have not search thorough enough
 Not believing in the significancy of the
research or overestimating its relevance
17/07/2015
INAOE
6
Dos and Don’ts
 Your thesis is NOT a core-dump of all you know about
everything [BundyA2004]
 …also it is helpful to create your own vademecum
 Be concise and go straight to the point, yet giving enough
background to make it self-contained
 Your thesis hang together in a coherent manner
[BundyA2004]
 Acknowledge:




17/07/2015
Everyone that has helped
Every institution involved
Every funding source
Every other’s work – avoid plagiarism
INAOE
7
Becoming an expert on a topic
 Read literature (both strictly and loosely related, although to a different
depth)
 …then read more …and more …and a bit more
 …not yet there…
 Talk to people
 …that include experts (of course!) but also non-experts (they are usually very good at
spotting holes!)
 Get feedback as well as new ideas
 Clearly bound your research
 …and your experimentation
 Abstract vs concrete
 Too abstract and it is likely that you won’t deliver
 Too concrete and it is likely that the contribution is not significant
 Write/Give talks
 Explaining things to other people (whether orally or written) is a good way of realizing

17/07/2015
one own’s limitations
When you are a real expert you hardly ever need overusing jargon
INAOE
8
Some tips
 Keep (organised) record of everything;
 Notes of your ideas
 Notes on read papers
 Set up and results of your experiments, but also distractions,
artefacts, etc
 Documentation of programs and code (that is in computer
science)
 etc
 Focus on principles and foundations, not on implementation
and performance
 Never, ever delete a file/data
 Keep different versions
 This helps to remember, but also allows you to go back if
something goes wrong
17/07/2015
INAOE
9
Some tips
 Read and write!
 For a good understanding, reading a paper once is often not
enough.
 It is fine to fail, and publish it
 It is research, so you may guess what’s going to happen, but
cannot be sure.
 Publishing bias may nevertheless make difficult to publish
negative results.
 Admit ignorance
 …make questions, even if you think: (i) the expert is going to
laugh at you, or (ii) what you are asking is obvious for everyone
else.
 “Quien pregunta es tonto 5 mins., quien no pregunta es tonto
toda la vida” Spanish saying
17/07/2015
INAOE
10
HOW TO ELABORATE AN MSc
THESIS
17/07/2015
INAOE
11
How to elaborate a thesis




Definition: What is a thesis?
What has a thesis got to demonstrate?
How to choose a thesis topic
The actors




The student
The supervisor
The thesis committee
The panel




Developing the project
Periodic monitoring by the supervisor and the committee
Writing the document
Formatting
 How to write the thesis
 Legal authorship (Institution) and intellectual authorship
(student)
17/07/2015
INAOE
12
Some references for this section
 [Camacho2003] Camacho Mejía, Felipe;
Herrera Barrera, Armando; Guía para la
elaboración de una tesis (2003)
Universidad Autónoma del Estado de
México (UAEMex)
 [Sloman??] Sloman, Aaron “Writing a
thesis” University of Birmingham
 [??] “How to write a PhD thesis”
17/07/2015
Dr. Felipe Orihuela Espina
13
Definition of a thesis
 A thesis is a scientific text describing a an
in-depth description of a phenomenon
(that includes computing) together with a
set of solutions/hypothesis to yet unsolved
problems or questions about the
phenomena, and that further provides
extensive support and evidence to back up
any claims made. [Self definition]
17/07/2015
Dr. Felipe Orihuela Espina
14
HOW TO CHOOSE A THESIS
TOPIC
17/07/2015
INAOE
15
What has a thesis got to demonstrate?
 That the student is capable of:
 Carrying out innovative research by himself
 Criticism; both towards other’s work and to his
own work
 Planning, executing and finishing a long term
project
 Communicating knowledge and ideas
 …in an organised fashion understandable by nonexperts
 …yet without sacrificing accuracy
17/07/2015
Dr. Felipe Orihuela Espina
16
How to choose a thesis topic
 Proposed by the department or the supervisor
 Often as a result of a research project, a

necessity/research priority of the institution or sometimes
from a collaboration with an industrial partner
Pros:
 Saves some headaches such as questioning one-self what do I



do?
The problem definition, extension and coverage are often well
studied and defined
Motivation and justification is clear
There is already a researcher interested or involved, almost
certainly with expertise on the topic
 Cons:
 You might not love the topic
 You might not get on well with the supervisor
 You risk just following instructions, and not demonstrating
initiative (necessary for obtaining the degree)
17/07/2015
Dr. Felipe Orihuela Espina
17
How to choose a thesis topic
 Proposed by the student
 Often as a result of the student’s interests
 Pros:
 You are likely to love the topic
 You are free to find a supervisor which you want to
work with
 You are already demostrating drive and initiative
 Cons:
 There might not be any researcher willing to supervise
the thesis or with enough knowledge about the topic
 You’ll have to work on the problem definition,
motivation, etc
 Funding?
17/07/2015
Dr. Felipe Orihuela Espina
18
THE ACTORS
17/07/2015
INAOE
19
The actors
 The student
 The supervisor
 The thesis committee
 The panel
17/07/2015
Dr. Felipe Orihuela Espina
20
The student
 You! The one carrying out the thesis project
 Responsible for:







Solving all problems related to the thesis research
Carry out the experiments
Proposing the solutions
Writing and submitting the protocol, the thesis and any papers that may
result
Successful finalization of the research
 If something goes wrong it is your fault, so do not blame your supervisor.
The scientific quality of the work (legal)
Finding funding for his/her conference attendance
 Entitled to:
 Institutional support
 An adequate supervision
 That’s not mean babysitting
17/07/2015
Dr. Felipe Orihuela Espina
21
The supervisor
 Whether the one who originally offered the topic or
chosen by the student.
 That’s not your boss, just your companion!
 In every aspect of the research, you have the last word.
It’s your thesis, not his/hers.
 Responsible for:




Providing adequate guidance all throughout
Providing starting knowledge/context about the problem
The scientific quality of the work (moral)
Ensuring that the student finishes in time and form
 …despite the student’s best efforts not to…
 Entitled to:
 When appropriate, co-authorships in publications
17/07/2015
Dr. Felipe Orihuela Espina
22
The supervisor
 NOT responsible for solving anything at all about the
thesis.
 The student shall not assume that his supervisor knows

everything about the topic
Moreover, after a year or so the student ought to have
surpass his supervisor in knowledge about the topic.
 Often can recommend bibliography and references
 May help to a degree in the writing and proof reading


of the thesis
May suggest a publication strategy
May help in funding search for conference attendance
17/07/2015
Dr. Felipe Orihuela Espina
23
The supervisor
 The supervisor is responsible for the
scientific quality of the work, as in time it
will become part of his CV and
consequently linked to his/her reputation
 However
 This responsibility is only moral, not legal.
 The student is free to submit his work even
without the consent of the supervisor and it is
not obliged to comply with any of the
supervisor recommendations.
17/07/2015
Dr. Felipe Orihuela Espina
24
The thesis committee
 4 members of the faculty plus 1 external
 Not all will have expertise related to the thesis topic, this
offers a different point of view
 Ultimately chosen by the faculty,
 …but the student and (perhaps) the supervisor may
proposed alternatives
 Responsible for:
 Periodically evaluating the student progress
 Detecting weaknesses in the research
 Providing suggestions for amending deviations menacing
the finalization within time constraints
 Entitled to:
 Being provided with the advances with enough time
 …in English if Spanish is not their first language!
17/07/2015
Dr. Felipe Orihuela Espina
25
The panel
 Peers assessing the final workpiece
 Experts on the field
 Responsible for:
 Evaluating the final thesis
 Rejecting works which do not exhibit originality, or
do not reach scientific standards
 Entitled to:
 Being provided with the advances with enough
time
 …in English if Spanish is not their first language!
17/07/2015
Dr. Felipe Orihuela Espina
26
HOW TO WRITE THE THESIS
17/07/2015
INAOE
27
How to write the thesis
 Developing the project
 Periodic monitoring by the supervisor and
the committee
 Writing the document
 Formatting
17/07/2015
Dr. Felipe Orihuela Espina
28
Developing the project
 A coarse overview:
 Reading about the background
 Establish a calendar
 Tentative: if no special constraints applied. Admit deviations
 Imposed: when special project constraints applied. Does not
admit deviations.
 If in engineering; then budget proposal




17/07/2015
 Inc. materials and human resources
Launch hypothesis and analyse project requisites and
demands
Design and execute experiments
Analyse your data
Write your final thesis document
Dr. Felipe Orihuela Espina
29
Periodic monitoring
 The goal is to ensure that the thesis is finished in
due time and form guaranteeing minimal scientific
quality (and engineering if it is the case)
 Advances should be check periodically
 The most common way for this monitoring are the

bi/weekly meetings with the supervisor
Thesis committee meetings often take place every 6
months and are exceptional occassions to assess the
real progress of the thesis
 Other forms of monitoring include:
 Seminar giving
 Technical reports writing
 Periodic report writing as requested by the
programme
17/07/2015
Dr. Felipe Orihuela Espina
30
Periodic monitoring
 How often should I meet my supervisor?:
 It really depends on your needs…
 However:
 Do NOT allow your supervisor to avoid meeting
you for long periods
 Do NOT relax yourself and “forget” to meet you
supervisor for long periods
17/07/2015
Dr. Felipe Orihuela Espina
31
Periodic monitoring
 What should I take to/prepare the meetings?
 Read in depth about the specific topic to discuss that week

 Ensure that you have your questions ready
Organize any results (figures and tables) that you will show
 It is rude to do it at the expense of the time of your supervisor
 If you have become stuck onto a particular problem
 Do not tell your supervisor until you have at least seriously tried to solve it
 …but if really stuck, then spend the most of the meeting on this.
 Remember that your supervisor may not know the solution!
 Have a list of non-research issues that ought to be disscussed: admin,
conference attendance, scholarship problems, etc
 Do not expect your supervisor to remember from one meeting to the next you
current needs
 If presenting a document (paper, report, protocol, etc) be sure that the
draft is polished to your best
 It is rude to use your supervisor as a spellchecker or as a primary school teacher
who has to tell you that every sentence you write is correct
 Respect the time of your supervisor!
 …during the meeting, and out of the meeting
 If exceptionally arriving late then apologize in advance by mail/phone
 Do not go beyond your scheduled time; perhaps other students are waiting
17/07/2015
Dr. Felipe Orihuela Espina
32
Periodic monitoring
 Your supervisor should in turn:
 Read whatever you sent prior to the meeting
BEFORE the meeting
 Be up-to-date with your research
 Not impose his will or point of view; but
instead give his/her best advice and let the
student take the decision
 Tell the student if other meetings and
responsibilities comes in the way
 Reallocate time if necessary so that the student do
not miss his meeting
17/07/2015
Dr. Felipe Orihuela Espina
33
Writing the document
 Who is it written for?
 Scientific peers
 The thesis is a scientific document:
 Be precise and concise
 Effort to ensure efficient and effective communication
 Avoid colloquial laguage
 Be scrupulous on correctness (lexical, orthographical, syntax,
grammar, etc)
 Artistics licences are valid as long as they do not prevent
clear communication of ideas
 Long sentences are often a bad idea
 Take care of the flow of ideas
 The reader does not know:
 your nomenclature, nor your acronyms, etc
 the details of your research and specifically of your
experimentation. Provide enough details for replicability.
17/07/2015
Dr. Felipe Orihuela Espina
34
Writing the document
 Assume knowledge only within expectable limits
 Level of detail
 Basic concepts
 If truly trivial, then omit them
 …but remember that the reader knowledge does not match
exactly yours!
 Things which are trivial for you, may not be trivial for him.
 Advanced concepts
 If fairly known, mention and provide adequate references.
 Concepts developed in your thesis
 They ought to be perfectly/accurately defined, and if
necessary with proper mathematical formulation.
 Avoid explining then in several places
 …however readability may demand some repetition
17/07/2015
Dr. Felipe Orihuela Espina
35
Writing the document
 Level of details
 Experimental description
 The secret is simple: Replicability
 Every aspect of your experiment has to be replicable with absolute
fidelity by the reader.
Not replicable from your text, not good enough

 Often convenient to provide justification for arbitrary decisions.
 Mathematical development, proofs and
demonstrations
 Omit if they are trivial (although they almost never are)
 Explain them as they unfold; do not only state them
 A thesis is not better or more complex conceptually just because
it has more maths and looks like more formal
 …that may scare a bachelor but not your panelist, and in turn it may
actually irritate them
 Ensure that they are utterly correct and coherent thorughout the

17/07/2015
document
Appendix are a good place for long developments and proofs
Dr. Felipe Orihuela Espina
36
Formatting
 Compulsory: Most times, the institutions have

regulations about the formatting (inc. front
cover, style, fonts, interline space, margins,
paper size (and weight!), etc). Be sure you
comply with them.
Availability; We may not have always the
software we like. Use whatever you need to
guarantee the best possible presentation.
 Never excuse a bad presentation blaming the
software/hardware tool you have used!
 Portability: The format chosen has to be
easily portable to other formats.
17/07/2015
Dr. Felipe Orihuela Espina
37
Authorship
 Legal authorship (Institution)
 The institution is the legal responsible for the
results.
 The institution can legally protect the intellectual
property of the knowledge generated by the
research and/or commercially exploit the
research
 …but in turn any damage that your research may
cause is the institution’s responsibility
 Silly example: If the bridge falls, its not your fault, but the
institution!
 Any other (internal) researcher can continue your
work (e.g. use your data!) without asking for
permission
 Although it is considered polite to ask for it…
17/07/2015
Dr. Felipe Orihuela Espina
38
Authorship
 Intellectual authorship (student)
 As the original author of the thesis, the student
keeps the intellectual property
 The student is therefore entitled to (and in fact
more often than not, encouraged to) publish your
work
 After published, any other (external) researcher can
continue your work without asking for permission
 The student can self-cite himself without
committing plagiarism (but beware of selfplagiarism and salami slicing issues!)
 The student can continue its research on the
topic even after finishing its liase with the
institution.
17/07/2015
Dr. Felipe Orihuela Espina
39
MAIN PARTS OF A THESIS
17/07/2015
INAOE
40
Main parts of a thesis
 It is very important for understanding
this part, that you go back to the slides
of week 1 and ensure you understand
all the foundations regarding
experimentation and research
methodology
17/07/2015
INAOE
41
Research Questions
 Phenomenon
 In Immanuel Kant’s philosophy, a phenomenon is the object of an
sensorial experience.
 [Real Academia Española de la Lengua]
 “In philosophy, any object, fact, or occurrence perceived or observed. In
general, phenomena are the objects of the senses (e.g., sights and
sounds) as contrasted with what is apprehended by the intellect.”
 [Enciclopedia británica]
 An observable event
 [American Heritage Dictionary of the English Language]
 A phenomenon is any instance or observable event
 …directly through our senses or via instrumentation
 Or perhaps even indirectly via the observation of its consequences
[https://en.wikipedia.org/wiki/Phenomenon]
17/07/2015
INAOE
42
Research questions
 They represent gaps in the knowledge
regarding the phenomenon of interest
 They ought to guide your research
 All goals (main and specific) are collateral
consequences of them
 All experiments are driven to answer them
 All experimental hypothesis are stated to
(educatedly) guess about them
 All conclusions are stated to satisfy them
17/07/2015
INAOE
43
Research questions
 Arguably the most important part of your thesis
 A rather bad habit is to state them (just because
you’ve been told to), and ignore them the next
minute…
 Another bad habit is to think that they are
secondary to goals.
 …that means you are unable to distinguish a

17/07/2015
research from an engineering project.
And remember; engineering can advanced science;
but not all engineering makes scientific contributions.
INAOE
44
Research Questions
 All (well stated) research questions always
have to include:
 …whether explicitly or implicitly
 The independent variables
 E.g. the originating phenomenon, causes, factors, etc
 The dependent variables
 E.g. the (observable) consequence, effects, outcomes,
endpoints, etc
 The controlled variables
 E.g. the context, co-factors, etc
17/07/2015
INAOE
45
Goals; Main and specific
 The main goal states what is to be
achieved during the thesis
 It is a long term goal
 It may fall BEYOND the reach and limits of the
project
 A thesis may be part of a more ambitious or bigger
project
 …that’s why establishing the limits is so critical
 It should be stated in one or two paragraphs
at most.
17/07/2015
Dr. Felipe Orihuela Espina
46
Goals; Main and specific
 Main goal
 Here at INAOE they like it as:
 “Developing an algorithm that bla, bla, bla… and that is
competitive with the state of the art”
 That is only acceptable if you understand that’s a
collateral goal
 The real goal in science is to understand a
phenomenon or in other words to generate new
knowledge (that includes computer science!),
 …that is; to establish a relationship between dependent and
independent variables under given constraints (i.e. controlled
variables)
 …therefore developing/implementing (and the likes)
correspond to engineering goals
17/07/2015
Dr. Felipe Orihuela Espina
47
Goals; Main and specific
 Specific goals
 Short term goals
 They will be covered during the thesis
 They may include developing specific
tools/algorithms, etc as byproducts of your
research
 They often/should include validation as one of
them
 Each one of them has to be describable in at
most 1 paragraph
 It is not enough to state them, it is necessary to
describe them
17/07/2015
Dr. Felipe Orihuela Espina
48
Experiments and Methodology
 Experimental chapters are the
fundamental/main part of your thesis
 This is your work!
 In a classical scheme, each experimental
chapter is dedicated to answer one research
question
 Concomitantly, each chapter may comply with
some specific goal or afford a specific
contribution
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
49
Experiments and Methodology
 The emphasis of each chapter will however
depend on your thesis aim:
 Emphasize methodology if novel or scarcely used
 Emphasize experimental design and rigour if
exceptional
 Remember! Experiments have to be carefully designed
and planned even if this is not the goal of the chapter
 Emphasize the research hypothesis associated to
the question
 Emphasize the contribution of the chapter
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
50
Experiments and Methodology
 The emphasis of each chapter will
however depend on your thesis aim
(Cont.):
 Emphasize differences in experimental design
against other authors’ choices
 Emphasize optimization of resources if
applicable
 Sometimes a better experiment is easy to design
or foresee but practical limitations prevent their
implementation
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
51
Experiments and Methodology
 The emphasis of each chapter will
however depend on your thesis aim
(Cont.):
 Emphasize the description of the algorithms
and their parameterization if the experiments
involve a collection of simulations
 Emphasize the sequence of the experiments
and simulations if thie is relevant
 Do not assume that commutative does apply
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
52
Experiments and methodology
 For each experiment, you should clearly declare at least:
1. Its unambiguous relation with the RQs.
2. Its unambiguous relation with the goals
3. Its research hypothesis
4. Its experimental hypothesis (both the null and the alternative)
5. Its experimental design inc. experimental units, factors, grups,
treatments, randomization (if applies), etc
It independent, dependent and controlled variables
Its identified sources of bias and confounders, and the efforts
to minimize them
The intended analysis (a priori) or the executed analysis (a
poteriori)
Verification and validation efforts.
6.
7.
8.
9.

17/07/2015
Give priority to validity types over mechanisms
INAOE
53
Experiments and Methodology
 The description of your experiments ought to be:






scrupulous,
rigorous,
exhaustive
utterly/brutally honest
reproducible
 Non reproducible experiments are simply useless!
guaranteeing replicability of your results
 Non replicable results are simply invalid!
 E.g.: Fix and declare the seed if any random process is
simulated in the computer.
 This is what science is all about!
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
54
Experiments and Methodology
 The description of your experiments ought to
be (Cont.):
 Indicate not only the factors you have
manipulated (independent variables) but also
everything that is kept constant (controlled
variables)
 Painstakingly describe:
 Set up
 Sessions progression
 All pre-processing, processing and analysis of your
data, and this includes the statistics for hypothesis
testing
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
55
Experiments and Methodology
 There is a full theory about experiment
design…
 It ensures your experiments reduce bias, cost
and time
 It has a strong statistical connotation that at
the end of the day is an optimization problem;
aiming to solve your unknowns with minimal
“effort” measured in some way
 …make sure you understand it and apply it!
17/07/2015
©Dr. Felipe Orihuela Espina (2006)
56
THANKS, QUESTIONS?
17/07/2015
INAOE
57