*start geekiness*
Kira -- I have a PhD in psychology and teach research design. I'm not in the trenches as it sounds like you are -- but one of my jobs is to teach my students to be more critically informed about scientific research.

So, I'm curious, in your field/job what process you go through to determine what's "worth considering and what isn't." What are the important issues you consider?

(the rest of you can ignore us for a while unless you like the nitty gritty...)

I think, when deciding whether some research has validity or usefulness, it's important to remember that not all research is funded research (or directly funded research). Much of the published research is unpaid student-driven research. The design and implementation of experimental research and the writing of articles for professional journal publication is often part of the degree-earning process in many health sciences. When a student comes up with a research idea, even if it is of questionable "worth" to humanity, it might be approved by the instructor depending upon the class for which the research was done. If it's a methodology class, the choice of subject matter may not be very important to the instructor (or the student). Some students may choose deliberately odd research for many reasons (to test personal theories, to make a satirical statement or just frivolous self-interest in choosing a topic that will make the research more interesting to them personally).

Frivoulous research with no possible implication, or research that is a replication of many previous studies, isn't likely to be published in the most highly regarded professional journals, but if the topic or outcome is timely, interesting and the paper is well-written, it can happen (especially since "uselessness" is also in the eye of the beholder).

No research study, no matter how well-designed and lofty of purpose provides more than a glimpse of the big picture. It isn't very smart to make lifestyle changes based on the description of results from one study (especially when the report comes from a secondary source such as a women's magazine).

And yet some of the most odd and/or impractical research has yielded important and valuable information at least indirectly, and/or eventually by the questions it inspired other researchers to explore.

Some of the important issues that I consider when I see a research paper include the following. And when I read a report of a study in the media that looks interesting, I do my best to find the original so that I can quickly assess it:

1. Theory being considered and background of the researchers, including but not limited to the theory itself, possible bias of the researchers, academic credentials of the investigators.
2. Design of the experiment (if you will), including but not limited to: is the theory actually being tested, or is the study designed to prove the theory, the size of examined group, method of assessing results (measurable? by questionaire? reliance on self-reporting? is the evidence anecdotal?), inclusion of base-line groups (i.e. those given placebos vs medicine in question), double-blind or not, length of time of study.
3. Method of result analysis.
4. Conclusions drawn and recommendations made.
5. Suggestion of further research avenues illuminated by the study in question.
6. List of references included in the study bibliography.
7. In what journal(s) is the study published?
8. Has the study been peer-reviewed?
9. Who funded the study?

So, if a company has funded research into the efficacy of a product that they are promoting, and it has been tested on 20 subjects who are family members, and is published in a journal funded by the product company, and has not been peer-reviewed, I am less likely to be impressed by its results. Take for example, the studies published by weight loss pill manufacturers, who conduct their own tests in their own labs and publish the results in their own publications and aren't bound by scientific procedure as their pills are considered food supplements not drugs. On the non-health side, paranormal research which uses the National Enquirer and News of the World as references to support their theories is, IMHO, marginally credible. Or studies of the blindingly obvious that lead to no useful conclusions or recommendations, such as a recent Scientific American study that is looking at why dogs have difficulty speaking like humans (seriously) -- apparently it is because they can't move their lips or mouths like humans according to Gary Lucas, visiting scholar in psychology at Indiana University Bloomington (Globe and Mail, p L6, Friday June 19, 2009).

I understand that my list isn't comprehensive nor complete by any means, but it is a quick checklist that I can do in a couple of minutes to help me weed out the bumpf. You would be surprised at the amount of promotional material that I get in the mail urging me to use product X in my profession based on "scientific research". Studies and references are always included. So with time, this system has served me quite well, and if some products look promising, I research those further (consultation with peers, examination of related studies involving the product, Continuing Education seminars focusing on the product, and so on).

For an interesting look at the humor in science, you can check out
http://improbable.com/
which is a journal devoted to the offbeat in scientic research. In this month's issue, they publish the study that examines the following burning question:

“Do Overweight People Remove Their Shoes Before Being Weighed by a Doctor? Consecutive Study of Patients in General Practice,” Timothy Harlow, British Medical Journal, 1997, vol. 315, p. 1663. The author, at rhe College Surgery, Cullompton, Devon, U.K., reports:

Casual observation and discussion with colleagues led me to the hypothesis that patients who are overweight tend to remove their shoes before being weighed by their doctor. I thought that this action was probably an attempt to reduce the reading on the scales. I tested this hypothesis by measuring the body mass index of patients who needed to be weighed as part of their management and noting whether they removed their shoes unprompted. To my knowledge, no such study has previously been performed.

and the fascinating world of navel lint studies.


JMHO

Kira

Kira == I think the biggest difference between the research you look at and I look at is that much of the psych research I'm focusing on is peer-reviewed and, if funded, funded by governments grants, e.g., NIH.

I do ask my students to read a lot of research that appears in the news, and then I do have them pay attention to a number of those questions, though usually much of that info is missing. It's enough for me to have them identify the kind of research (e.g., correlational, experimental) and whether they have much info about the sample and methods. I focus a lot on measurement issues.

As for the number of participants, I am not bothered by small numbers, esp in experimental research with significant results. If the results are significant with small numbers, then there's frequently a large effect. Don't need more people to make results more 'significant'.

But the type of sample -- yeah, I can see how family members would be a problem!

Hi there!
I guess this is the problem that I have with most diet and health "research", because most of the stuff out there says, basically, "scientific studies show that...", or "according to research, x is bad for you..." or "those who ate X reported Y..." and you really have to be careful with that. If you aren't able to think critically, then you may not be getting the whole picture. If a study shows that eating 6 eggs a day is optimal, and the Egg Board funded and published the study, I'm going to have some questions. And I'll point this out in the thread. And I'm sure I'll be pm'd about this, as I have been in the past...

I see your point with sample size. I guess the only thing I point to is the issue of extrapolating results from small sample populations to the general population. Take, for example, cancer study results. If 20 cancer survivors display result x after treatment y, for example, it leads to a study involving long term, high population, and geographically distant subjects. This is to minimize the possibilities of geographical influence and factors specific to these 20 people (i.e. is it the treatment? Or is it something in the geographical area that produces the result?), rather than a widespread immediate change in treatment philosophy. It is really hard to make an appropriate scientically-based recommendation that affects millions if a small study sample is involved. But small population study IS in turn valuable as it is easier to isolate factors that may be influential (why are there more centenarians in Okanawa than anywheres else in the world, for example).
It DOES go both ways. But for my profession, if a study comes out that has a very small study population but the results are really significant, I'll tend to wait for further research before recommending or using product X unless there is a super-compelling reason...



Kira

ps I'll bet you didn't think I really was a doctorate! You never know on the internet

Know what "iatrogenic" means? It literally means "doctor-caused"

I'm another one who works in the science sector; I type up a lot of manuscripts and submit them to journals. One thing that might be good to know is that just because something is published in a medical or scientific journal, that isn't holy writ on the topic; it may barely be the first word, in effect the authors are inviting other researchers to see if they can duplicate the result. That article in The Lancet claiming a connection between the MMR vaccine and autism was just a preliminary report; it's since been thoroughly discredited (let's not turn this thread into a holy war on what does or doesn't cause autism, pretty pretty please with sugar on top, huh?).

That said: hey, the OP was making a joke; do we have to wet-blanket it by taking it so seriously?

PS: Yeah, I take off my shoes when I weigh in @ the doctor's office

Kira -- I missed your post from a few days ago. Thanks for the thoughtful response.

I completely get your point about generalizability. I guess I was talking about effect sizes. For me, large sample sizes don't really address the issue of generalizability, which could really only be addressed in a single study with a truly random sample, and I don't know of much (any?) clinical research that gets one of those. I completely agree with the point both you and Another are making. ANY single study is NEVER definitive, and frequently preliminary.

I think lots of people don't know that. They see scientific results and think "truth" not "on a long rambling path to knowledge..."

Kira

I think we chased off everyone else... sorry!

Too funny! But I LOVED the discussion and I LOVE this site for allowing this kind of discussion, because it really IS important.
Thanks for indulging me!
Kira

Given your background, I'm sure you don't mean that the way it sounds to me, so I'm just going to clarify for others. Sample size very much does matter to make the results significant. If your sample size is too small, you simply cannot prove that the results are not due to chance alone. If you toss a coin twice, and you get heads both times, that will seems significant. It seems like heads are a very likely result. But it's not. It is due to chance, and the next three tosses may all be tails.

Julie -- Let me clarify. First, what I meant by significance relates to the use of inferential statistics such as t-tests, ANOVAs, correlations. For these tests, when a result is significant, we do mean, as you note, that the results are not due to chance.

Significant results can occur with a small group of people. Let's say I do a simple experiment with my class of 14 students, a memory study where 1/2 get one set of directions and 1/2 get another. At the end, I tabulate the results, conduct a t-test and find that one group did 'significantly' better than the other, meaning that this result is not due to chance. Actually, I do something like this every term. On a memory test worth up to 40 points, I often get results where 1 group scores twice as well as the other... maybe better. This is what I would call a large effect -- the only kind I am consistently likely to find with such a small sample. Getting a significant result with a small sample is great. In that case, getting a larger sample isn't going to make the results 'more' significant.

Now, this doesn't mean that my results can generalize beyond the kind of sample I used, or generalize beyond the very specific method I used.

So, I'm not saying sample size is never important, frequently you do need larger sample sizes to demonstrate significant results. But if you have a small sample and have generated significant results, it's not a valid criticism to say you should have a larger sample.

I hope that helps...

I second this ^^

Virtually all of my experiments use group sizes of 8. This means that in order to find significant differences, the effect sizes have to be somewhat large and/or the variability within groups has to be low. But this is actually a good thing, believe it or not. If I increased my group sizes, it would allow me to detect really tiny effects, sure, but just because an effect is statistically significant doesn't necessarily make it "meaningful" or "important."

For example if treatment X yields a mean of 59.4 in group A while the control group comes in at 59.6, this difference might be found to be statistically significant if the sample size is big enough and the variability is low enough. For my purposes, though, such a tiny difference wouldn't justify me further pursuing this line of research or publishing the data.

Of course, I work with inbred rats, not humans, so I can afford to keep the group sizes small (and I should, for the reason mentioned above). I just wanted to point out that group size alone doesn't make the results of a study more or less credible.

Drina -- Thanks for mentioning the distinction between "statistically significant" and "meaningful or important." They are not necessarily the same thing!

What research do you do with rats, btw?

This is also quite true and another reason a great deal of "evidence" thrown around in the popular press is rubbish. This is why labels say things like "Helps" burn fat, or the like. It might be significant to the same degree as adding a drop of water to the ocean, but they don't say that. Just that it "helps."