Mountain Project Logo

Any statisticians out there?

Original Post
Mark E Dixon · · Possunt, nec posse videntur · Joined Nov 2007 · Points: 974

I'm looking for a ballpark impression. I don't have access to the original data.

But here's the situation. You have an experimental group with either 4 or 5 subjects. The mean for the initial state is 47 with a standard deviation of 15. The mean post intervention is 56 with an sd of 14.

Is it conceivable that the effect size is 0.7? Just seems way too high for me, but it's statistics and all...

Larry S · · Easton, PA · Joined May 2010 · Points: 872

Not a statistician... but see if the confidence intervals for the population mean overlap... m.dummies.com/how-to/conten… if they do I don't think you can say much of anything. You also don't know anything about the populations. See here en.m.wikipedia.org/wiki/Ans…

Buff Johnson · · Unknown Hometown · Joined Dec 2005 · Points: 1,145

Did you t-test it?

StatJuan · · Unknown Hometown · Joined Jul 2013 · Points: 10

With those sample sizes and variability that size, just about anything is possible. Lower bound on a 95% CI for the true difference in means is roughly -15 (based on normal theory, which can't be validated without the data, or honestly even with the data with those sample sizes, and I'm using the standard error coming from a Welch's t-test but have no interest in computing the degrees of freedom for the Satterthwaite approximation). So it's all very back of the envelope. To put it another way, if we ran a hypothesis test for the difference in the true means equal to .7 vs a 2-sided alternative, the p-value based on this data (and sample sizes of 4 for both) as .54. If I assume n=5 for both, the p-value only drops to .49.

Mark E Dixon · · Possunt, nec posse videntur · Joined Nov 2007 · Points: 974
StatJuan wrote:With those sample sizes and variability that size, just about anything is possible. Lower bound on a 95% CI for the true difference in means is roughly -15 (based on normal theory, which can't be validated without the data, or honestly even with the data with those sample sizes, and I'm using the standard error coming from a Welch's t-test but have no interest in computing the degrees of freedom for the Satterthwaite approximation). So it's all very back of the envelope. To put it another way, if we ran a hypothesis test for the difference in the true means equal to .7 vs a 2-sided alternative, the p-value based on this data (and sample sizes of 4 for both) as .54. If I assume n=5 for both, the p-value only drops to .49.
That is so far over my head it isn't funny.

Bottom line, maybe not a conclusive result?
Aleks Zebastian · · Boulder, CO · Joined Jul 2014 · Points: 175

climbing friend,

May I suggest you get a girlfriend?

Mark E Dixon · · Possunt, nec posse videntur · Joined Nov 2007 · Points: 974
Aleks Zebastian wrote:climbing friend, May I suggest you get a girlfriend?
My wife might object.
Steven Lee · · El Segundo, CA · Joined Mar 2014 · Points: 385
Mark E Dixon wrote: That is so far over my head it isn't funny. Bottom line, maybe not a conclusive result?
Definitely not an expert, but here's how someone explained it to me. Maybe it'll help?

Null hypothesis states there is no difference between the two populations. P-value would be the probability you'd get the same result if there were no difference in the population.

Basically - you're hypothesis in this case is that there's a difference between the two sets of data. If there actually were no difference, the probability of getting the same result is ~50%. So basically 50% of the time, you will have incorrectly identified the truth.

So pretty much not a conclusive result. However, StatJuan made a lot of assumptions. To even use the t-test, you need to have normal data which you don't know.
Erick Valler · · flat midwest · Joined Feb 2011 · Points: 20

Being a nerd (that is a trained brewer) and finding that most people posting here are nerds in general (and like drinking beer after a great satisfaction of the flash), a fact about the Student T Test. the t test is a statistical test developed by a lab technician/employee of the Guinness brewery (way back when). After being forbidden to publish under their own name they labeled their method the "student" t test for publication purposes. This, as well as many other fundamentals, are founded in brewing science. Hope this helps others think deeper about the $1.50 beer they end their glorious day of sending with. Or you're one of the 3:45am dingbats that read this immediately upon posting and made me realize I'm about $25 deep already...

Tev · · Hickory · Joined May 2012 · Points: 25

How about a regression-phase diagram?
Did you know that 69% of all statistics are made up?

Buff Johnson · · Unknown Hometown · Joined Dec 2005 · Points: 1,145

Dyk 3 out of 2 people have trouble w/ fractions?

El Duderino · · Unknown Hometown · Joined Feb 2013 · Points: 70
Mark E Dixon wrote:I'm looking for a ballpark impression. I don't have access to the original data. But here's the situation. You have an experimental group with either 4 or 5 subjects. The mean for the initial state is 47 with a standard deviation of 15. The mean post intervention is 56 with an sd of 14. Is it conceivable that the effect size is 0.7? Just seems way too high for me, but it's statistics and all...
So technically you should be running a paired t test given that it sounds as though your samples are not independent (you took individual A and measured some metric, applied some treatment, and then remeasured that same individual post-application). In that case, you really need the standard deviation and mean of the differences of individual data points(ie calculate the difference in the values for Apre and Apost, Bpre and Bpost, etc, average these differences, and then find the SD of the differences).

However, I ran both a paired T test under the assumption that the difference in the means was equal to the mean of the differences and that the SD was roughly equal and a 2 sample t test (violating the independence assumption). Both gave me p values of >0.10 (=0.31 and 0.42, respectively). P values are the probability that you collected this data given the null hypothesis (in this case that the two groups are the same and that the treatment has no effect) is true. Basically, p values greater than 0.1 or 0.05 mean that you can't reject the null hypothesis (ie, you can't tell the two groups apart statistically). Given those values, the two groups are too similar ( iopimedical.com/Images/Lip%…) and any differences are probably just due to random error.
StatJuan · · Unknown Hometown · Joined Jul 2013 · Points: 10
scienceguy288 wrote: and any differences are probably just due to random error.
The probability that the difference is due solely to random error is either 0 or 1. You are accepting the null with this language, which doesn't seem like a good idea when the sample means are different by 9.
Buff Johnson · · Unknown Hometown · Joined Dec 2005 · Points: 1,145

Need more n

El Duderino · · Unknown Hometown · Joined Feb 2013 · Points: 70
StatJuan wrote: The probability that the difference is due solely to random error is either 0 or 1. You are accepting the null with this language, which doesn't seem like a good idea when the sample means are different by 9.
One cannot accept the null, if you want to get pedantic. One simply cannot reject it to accept the alternate hypothesis. Yes, the sample means are different by 9, but if the SD is as high as he suggests, the distribution of randomly sampled data would overlap a lot imgur.com/uQZYvy1 and one can't say that group a differs from b because of treatment. The difference that is observed may be just due to random sampling error and variation within the population. Of course, this assumes a normally distributed distribution, which we can't assess...
Mark E Dixon · · Possunt, nec posse videntur · Joined Nov 2007 · Points: 974
scienceguy288 wrote: this assumes a normally distributed distribution
Almost certainly abnormal, given that the subjects are climbers :-)

Thanks to all who have contributed, even though my head is kind of hurting now trying to understand the posts.
El Duderino · · Unknown Hometown · Joined Feb 2013 · Points: 70
Mark E Dixon wrote: Almost certainly abnormal, given that the subjects are climbers :-) Thanks to all who have contributed, even though my head is kind of hurting now trying to understand the posts.
TL;DR, since we can't see the data, it's hard to tell, but the difference probably isn't significant given what we have...
StatJuan · · Unknown Hometown · Joined Jul 2013 · Points: 10
scienceguy288 wrote: One cannot accept the null
One can do whatever they want. One can even say things like 'any difference is probably just due to random error" though I would advise against it, as it sounds like you are accepting the null.
StatJuan · · Unknown Hometown · Joined Jul 2013 · Points: 10
polloloco wrote: However, StatJuan made a lot of assumptions. To even use the t-test, you need to have normal data which you don't know.
Which I was very clear about. And no, you don't need normal data to use the t test, you need the sampling distribution of the estimator to be normal. Unless you want to argue that the required independence between x-bar and s^2 only holds for normal data. But if we worried about this, we could never use the t-test because there is no such thing as normal data.
Steven Lee · · El Segundo, CA · Joined Mar 2014 · Points: 385
StatJuan wrote: Which I was very clear about. And no, you don't need normal data to use the t test, you need the sampling distribution of the estimator to be normal. Unless you want to argue that the required independence between x-bar and s^2 only holds for normal data. But if we worried about this, we could never use the t-test because there is no such thing as normal data.
Just re-read my comment, and it came off like I was critiquing - which is poor wording on my part. Sorry, I definitely did not mean that. You were very clear about the limitations and I was just repeating it for emphasis.

I'll have to let the comments on the sampling distribution, x-bar, and s^2 of the estimator marinate in my head to understand it. Thanks for the new knowledge! MP... maybe we need a nerdout section, haha.
Doug Hemken · · Madison, WI · Joined Oct 2004 · Points: 13,668

Mark, given your description of the design, "effect size" is likely based on paired differences? So the pre- and post- summary statistics don't really tell the story ... that's the whole reason paired differences is useful.

I take it effect size is mean(paired differences)/sd(paired differences) ?

Guideline #1: Don't be a jerk.

Training Forum
Post a Reply to "Any statisticians out there?"

Log In to Reply

Join the Community

Create your FREE account today!
Already have an account? Login to close this notice.

Get Started.