ATC or Grigri? Official MP survey

Ever been dropped? Ever dropped someone? Collecting some data for a Mountain Project analysis to be featured on the blog in the next month or so. Goal: determining once or for all which is safer in the hands of an average user: an ATC or a grigri. Anonymous 60-second survey below:

docs.google.com/forms/d/e/1…

You really think that a survey of people who self-select into taking your survey from a forum post has any validity whatsoever? This doesn't even pass the most generous sniff tests for survey validity, and serves only to mislead.

You want to answer this question for real? PM me, and we can talk. In the meantime, take the "official" down before you embarrass yourself.

There are mountains of scientifically collected data on this that are a much better basis for drawing a conclusion. I'll save you the Googling: Grigris are, by far, safer. Their accidents are just over reported because they're so surprising.

Needs a comparison for how often each style is used.

E.g. I only leave the ground if belayed with a grigri, cinch, or jul, so if a user like me were to log a report the data is skewed without your dataset having a variable that you can use to correct for which device I choose to use

Ted, that's what I'm trying to find out. If you have access to more scientifically collected data, please PM me; I'd love to see it! However, I've been unable to find any conclusive evidence for the safety of one device over the other (outside the gym) as of the posting of this survey.

Shoo, I agree that survey respondents will be self-selecting in that they're more likely to respond if they've been dropped, but I have no reason to believe Mountain Project users would be predisposed to use a Grigri over an ATC or vice versa. Therefore, a comparison of the two should still be valid if corrections for a small sample size are applied. And though any evidence will still anecdotal, some numbers would shed more light than current forum discussions do. This is also meant to be used in conjunction with other sources, not as standalone evidence. If you have any more concerns, please feel free to PM me; I appreciate your feedback.

Medic741 - That's an excellent point. Please mention that in the comment/clarification section of the survey, if you haven't already. I'll see if I can add another question.

CoreyLynn wrote:Shoo, I agree that survey respondents will be self-selecting in that they're more likely to respond if they've been dropped, but I have no reason to believe Mountain Project users would be predisposed to use a Grigri over an ATC or vice versa. Therefore, a comparison of the two should still be valid if corrections for a small sample size are applied.

You are both wrong about your assumption that MP users (and moreso MP users reading a very specific sub-forum who bother to click your link) will not differentially respond based on own experience AND ALSO missing a hugely important second set of factors that are completely unaccounted for.

1) This is a charged issue, particularly for readers of this forum. People feel very strongly about "dangers" of the grigri. You are going to get a huge amount of people who are out to prove either that grigris are dangerous / totally safe based on their anecdotal experience. Your problem is that you have no way of knowing the relative strength of those biases. Some of those biases are in opposite directions.

You are making the mistake of believing that ambiguous net direction of bias strengthens conclusions, when the opposite is in fact true. If you know you are biased, but don't know what direction you are biased in, you cannot make any conclusions. If you know what net direction the bias is expected to be, you have a chance that the data will come out in such a way that some conclusions can still be made.

2) You are completely failing to understand the differences in how the two devices are used. The kinds of people and scenarios in which people use the two devices are totally different, and not in a way that is easily accountable (nor do you even attempt to do so with a BS measure of "years of climbing"). These are NOT fair comparisons. Again, you have an ambiguous direction of bias problem, complicated with the self-reporting biases outlined above, although in this case one general direction of bias is likely to be a bit stronger overall.

3) Forget about the "small sample correction" nonsense. That is not even close to your biggest problem here.

CoreyLynn wrote: And though any evidence will still anecdotal, some numbers would shed more light than current forum discussions do

No. Bad data masquerading as good data serve no useful purpose in public discourse. Your results are meaningless, and are actively misleading except in the absurdly unlikely case where all of the numerous biases in your survey magically cancel each other out.

Again, there are good ways to do this and actually put some real numbers on this, but you are not doing any of them. You have good intent, but you are doing a bad thing. Please cancel this.

Is this for a school project?

CoreyLynn - I think it absolutely fantastic that you are doing this survey.

It seems the trick is you need a common denominator for comparing the two devices-- and this denominator is really hard to get. Like number of drops per number of falls. Looks like you are going to do drops per years of climbing. But then there is the factor that people who are catching lots of falls (sportclimbers) tend to use a grigri. Maybe people could estimate the number of falls they catch per year.

Shoo - sooo, rather than just take aim at this survey, how would you set up a survey that, in your **opinion**, is not biased?

Don't listen to Shoo: his comments and opinions are as meaningless as your internet survey may ultimately be. It's a MP-forum discussion after all. Nonetheless it's interesting data, and I enjoy seeing what people are using.

Surprised that twice as many people have reported being dropped on ATC vs Gri Gri. Well I mean not really but I didnt think it would be that high.

When your life depends on it, GriGri

Nothing to raise people's hackles like calling something official for no reason other than to sound important...

Also, where's the love for Smart/clickup/megajul style devices?

Wish there was a hear-say category...

To the 3 people who have claimed to been dropped 10 times by their belayer... what?!

EthanC wrote:Nothing to raise people's hackles like calling something official for no reason other than to sound important... Also, where's the love for Smart/clickup/megajul style devices? Wish there was a hear-say category...

It worked for the Federal Reserve and Federal Express, neither of which are federal.

JohnnyG wrote:Shoo - sooo, rather than just take aim at this survey, how would you set up a survey that, in your **opinion**, is not biased?

Basically any sampling strategy that relies heavily on self-selection into the sample is out, pretty much automatically.

Getting a real answer to this question would require some leg-work in the best case scenario, with reductions in bias depending strongly on just how much leg-work you're willing to do. An absolutely minimal amount of work would probably require cold call phone polling from a "climber" database (AMC, gym records, etc).

I am happy to do some pro bono work designing a study here, but my professional/research associations would prevent me from from doing much more than design and advising.

Ancent wrote:Don't listen to Shoo: his comments and opinions are as meaningless as your internet survey may ultimately be.

My "meaningless opinions" have credentials to back them, and you're almost certain to get the same answer from anyone with a relevant/comparable background. Once again, PM me if you want more info.

Shoo is right in terms of survey design. This is not a sample survey that can be generalized to any population. It is only a questionnaire of those who respond.

And yeah, those people with 10 drops need to find a different hobby.

shoo wrote: my professional/research associations would prevent me from from doing much more than design and advising.

I wouldn't argue with this man about stats... sounds like you'll loose 100% of the time.

Owen Witesman wrote:Shoo is right in terms of survey design.

Agreed. Is the survey for personal interests? Or part of something larger?

Cory F wrote: Agreed. Is the survey for personal interests? Or part of something larger?

You've asked twice now, but CoreyLynn says what the info is for in the originating post...

shoo wrote: Basically any sampling strategy that relies heavily on self-selection into the sample is out, pretty much automatically. Getting a real answer to this question would require some leg-work in the best case scenario, with reductions in bias depending strongly on just how much leg-work you're willing to do. An absolutely minimal amount of work would probably require cold call phone polling from a "climber" database (AMC, gym records, etc). I am happy to do some pro bono work designing a study here, but my professional/research associations would prevent me from from doing much more than design and advising. My "meaningless opinions" have credentials to back them, and you're almost certain to get the same answer from anyone with a relevant/comparable background. Once again, PM me if you want more info.

I see your point, and I agree with you.

This gets to really basic questions about data and decision making. I don't think it's appropriate to completely disregard this study because of self selection, especially since the alternatives is a much more costly study.

I work with a lot of imperfect data as a geologist, and we make decisions on imperfect data in all fields. Basically, we have to make decisions whether there is perfect data or not. And I would argue that some data is better than no data.

JohnnyG wrote:And I would argue that some data is better than no data.

Some data is better than no data for those that have the education / training / experience / time / attention span to understand the limitations and biases of those data. Unfortunately, the typical MP user does not have those luxuries.

It is 2016. It is easier than ever to collect and spread information, good or bad. If it is more likely that the information that will be spread is misunderstood than understood by those who would be making decisions on that data, then no data is better than some data.

Knowing this, it is irresponsible, and possibly downright dangerous, to produce shitty stats and endorse them with an otherwise respectable brand, no matter how many disclaimers you use.

Brett, I'm an idiot. I read the OP before coffee. embarrassing...