Verizon first in Consumer Reports satisfaction survey

"Bob Smith" <[email protected]> wrote in message
news:yBlOb.12602

> > 31,000 out of over 70 million cell phone subscribers is very big? Its
> less
> > than .05%, and given the way the survey was conducted, much too small to
> > have any legitimacy.
>
> I'd agree with Scott.

The margin of error relates strictly to the sample size. This was a huge
sample size. Do the math yourself. It's one over the square root of the
sample size (multiply by 100 to convert to a percentage)

> What's more, with the survey, how followed up to make sure that what was
said was true.

This was a survey of subcribers, based on their experiences with their
carrier. In every survey a few people may lie, but with such a huge sample,
a few liars are inconsequential. And of course there is no reason to believe
that subscribers from one carrier lie any more or less than subscribers from
another carrier. You can't follow up on a survey to ask if what respondents
said is true was actually true. You have to look to see if there would be
any motivation for lying.

>BTW, who's doing the independent audit on CR? I don't trust one thing they
say ...

You may not agree with their own evaluations of products for whatever
reason, and I often feel this way too. But their surveys of owners of
products and services are very valuable because they always use such large
sample sizes.

---

› See More: Verizon first in Consumer Reports satisfaction survey

"Steven M. Scharf" <[email protected]> wrote in message
news:[email protected]...

>
> The margin of error relates strictly to the sample size. This was a huge
> sample size. Do the math yourself. It's one over the square root of the
> sample size (multiply by 100 to convert to a percentage)

The margin of error is not the bone of contention here. The survey
represents less then .05% of all cellular users, and no effort was made to
equally represent all carriers, or insure that an equal percentage of users
from each carrier was included. For all we know, this was a survey of
30,000 Verizon users, 500 ATT users, 300 Cingular users, 100 T-Mobile users,
80 Nextel users, 10 Sprint users and 10 users of other carriers. The
controls, and the way the users were gathered, leave too much room for
inaccurate information.

>
> > What's more, with the survey, how followed up to make sure that what was
> said was true.
>
> This was a survey of subcribers, based on their experiences with their
> carrier. In every survey a few people may lie, but with such a huge
sample,
> a few liars are inconsequential. And of course there is no reason to
believe
> that subscribers from one carrier lie any more or less than subscribers
from
> another carrier. You can't follow up on a survey to ask if what
respondents
> said is true was actually true. You have to look to see if there would be
> any motivation for lying.
>
> >BTW, who's doing the independent audit on CR? I don't trust one thing
they
> say ...
>
> You may not agree with their own evaluations of products for whatever
> reason, and I often feel this way too. But their surveys of owners of
> products and services are very valuable because they always use such large
> sample sizes.
>

..05% of a user base is not a large sample.

"Steven M. Scharf" <[email protected]> wrote in message
news:[email protected]...
> "Scott Stephenson" <[email protected]> wrote in message
> news:iNidnfM0-
>
> > 31,000 out of over 70 million cell phone subscribers is very big? Its
> less
> > than .05%, and given the way the survey was conducted, much too small to
> > have any legitimacy.
>
> Yes, statistically it is very large. This large of a sample has a margin
of
> error of less than 0.6%, much smaller margin of error than most surveys.
You
> can read about sample sizes and margins of errors at:
> http://www.robertniles.com/stats/sample.shtml
>
> There is no reason to believe that of the people that responded to the CR
> survey, the ones with Verizon conspired to give it good grades, while AT&T
> and Cingular subscribers conspired to give their carriers bad grades.
>
> The one thing that CR does very well is surveys. You may disagree with
their
> opinions on the most important characteristics of automobiles or
> dishwashers, but when they do surveys on products, you can be sure that
the
> results are accurate.

Accurate for the sample surveyed, which is never more than a survey of the
same customer base time after time. There is never an effort to go outside
of CR subscriber base (which does not ever represent the average consumer),
and the use of the same static base for all of their surveys renders all
data useless- you are getting the opnions of the same group of people every
time.

"Scott Stephenson" <[email protected]> wrote in message
news:W_-dnSxfqviDm5fdRVn-

> The margin of error is not the bone of contention here. The survey
> represents less then .05% of all cellular users, and no effort was made to
> equally represent all carriers, or insure that an equal percentage of
users
> from each carrier was included. For all we know, this was a survey of
> 30,000 Verizon users, 500 ATT users, 300 Cingular users, 100 T-Mobile
users,
> 80 Nextel users, 10 Sprint users and 10 users of other carriers. The
> controls, and the way the users were gathered, leave too much room for
> inaccurate information.

Sorry, you're incorrect. The minimum number of respondents per city per
carrier was 150. The controls were in place to ensure that the data was
accurate. Even at the minimum response level the margin of error is low.

Also, remember that this was not a survey where 31,000 people told CR who
their favorite carrier is. This survey was asking cellular customers from to
rate the service of their own carrier.

> .05% of a user base is not a large sample.

You're correct. It's not large, it's huge. Even if you divide up the total
number of respondents by metro area, it's still a large sample per metro
area, statistically speaking.

As the article stated, the ratings are based strictly on survey results. Now
if you have some reason to believe that Consumer Report subscribers (of
which I am not one, BTW), are more likely than the population at large to
rate a specific carrier better or worse, I'd love to see that evidence.

I do believe that the margin of error for Verizon was probably less than
that of other carriers, simply because Verizon is the largest carrier and no
doubt got the most responses. Furthermore, since CR subscribers tend to be
better educated and higher income than the population at large, I'd expect
that there are more Verizon subscribers among CR subscribers than the
population at large. However this does not change the actual ratings.

"Scott Stephenson" <[email protected]> wrote in message
news:8ZmdnWEUZs-

> Accurate for the sample surveyed, which is never more than a survey of the
> same customer base time after time. There is never an effort to go
outside
> of CR subscriber base (which does not ever represent the average
consumer),
> and the use of the same static base for all of their surveys renders all
> data useless- you are getting the opnions of the same group of people
every
> time.

If the survey was "who's the best cellular carrier in xyz area?" you'd be
correct. However this was not how the survey was set up. None of the data is
normalized. It's really 62 separate surveys (4-6 carriers per city), each
with a minumum sample size of 150. It's asking Verizon subscribers to rate
Verizon, AT&T subscribers to rate AT&T, and so on. It's not asking Verizon
subscribers what they think of AT&T, etc.

You'd have to claim that CR subscriber is more (or less) likely to rate a
carrier better (or worse) than the general public. Even if this were true,
the overall ratings would still be the same in relative terms (unless you
claim that the CR subscriber's bias would vary by carrier).

Actually the margin of error is higher than what I stated earlier. With the
minimum of 150 responses, the margin of error per survey could be as high as
8%, though this would be in very few cases, and probably only for T-Mobile
or Nextel.

"Steven M. Scharf" <[email protected]> wrote in message
news:[email protected]...
> "Scott Stephenson" <[email protected]> wrote in message
> news:W_-dnSxfqviDm5fdRVn-
>
> > The margin of error is not the bone of contention here. The survey
> > represents less then .05% of all cellular users, and no effort was made
to
> > equally represent all carriers, or insure that an equal percentage of
> users
> > from each carrier was included. For all we know, this was a survey of
> > 30,000 Verizon users, 500 ATT users, 300 Cingular users, 100 T-Mobile
> users,
> > 80 Nextel users, 10 Sprint users and 10 users of other carriers. The
> > controls, and the way the users were gathered, leave too much room for
> > inaccurate information.
>
> Sorry, you're incorrect. The minimum number of respondents per city per
> carrier was 150. The controls were in place to ensure that the data was
> accurate. Even at the minimum response level the margin of error is low.

The more correct way to take the survey would have 150 respondents per
carrier, or x% of a carrier's subscriber base. The 'controls' you mention
make my scenario more likely to occur, although not in the very sarcastic
way I proposed it.

>
> Also, remember that this was not a survey where 31,000 people told CR who
> their favorite carrier is. This survey was asking cellular customers from
to
> rate the service of their own carrier.

Correct. But a skew in the number of respondents per carrier skews the
results.

>
> > .05% of a user base is not a large sample.
>
> You're correct. It's not large, it's huge. Even if you divide up the total
> number of respondents by metro area, it's still a large sample per metro
> area, statistically speaking.
>
> As the article stated, the ratings are based strictly on survey results.
Now
> if you have some reason to believe that Consumer Report subscribers (of
> which I am not one, BTW), are more likely than the population at large to
> rate a specific carrier better or worse, I'd love to see that evidence.
>
> I do believe that the margin of error for Verizon was probably less than
> that of other carriers, simply because Verizon is the largest carrier and
no
> doubt got the most responses. Furthermore, since CR subscribers tend to be
> better educated and higher income than the population at large, I'd expect
> that there are more Verizon subscribers among CR subscribers than the
> population at large. However this does not change the actual ratings.
>

Assuming that Verizon has a more intelligent subscriber base is pure crap.
But thanks for pointing out the biases of a CR survey in that last
paragraph- it further invalidates the data as presented. Myjob here is
done.

"John Richards" <[email protected]> wrote in message
news:[email protected]...
> Steven Scharf wrote:
>
> > 31,000 responses is a pretty good sample size. No reason to doubt the
> > validity of this survey.
>
> My sentiments exactly.

Technically the survey is actually 62 separate surveys (4-6 per city).
They're not asking subscribers: "who's the best carrier?" They're asking
Verizon subscribers to rate Verizon, Sprint customers to rate Sprint, etc.
And they don't include results if there were less than 150 respondents per
carrier per metropolitan area. But with 31,000 respondents, they'll have an
average of around 500 respondents per carrier per city. Still a valid
sample.

Looks like some people are upset that their carrier did poorly, so they try
to question the validity of the survey, but they have no basis for this.

"Steven M. Scharf" <[email protected]> wrote in message
news:[email protected]...
>
> Looks like some people are upset that their carrier did poorly, so they
try
> to question the validity of the survey, but they have no basis for this.
>
>

Looks like you don't know what you are talking about. Actually, my carrier
did quite well- better than most. But that doesn't make the results any
more believable or accurate. I could post the questions to this and every
other newsgroup, and by your logic, the data would be just as valid if I put
two controls in place.

"Scott Stephenson" <[email protected]> wrote in message
news:[email protected]...
>
> "Steven M. Scharf" <[email protected]> wrote in message
> news:[email protected]...
>
> >
> > The margin of error relates strictly to the sample size. This was a huge
> > sample size. Do the math yourself. It's one over the square root of the
> > sample size (multiply by 100 to convert to a percentage)
>
> The margin of error is not the bone of contention here. The survey
> represents less then .05% of all cellular users, and no effort was made to
> equally represent all carriers, or insure that an equal percentage of
users
> from each carrier was included. For all we know, this was a survey of
> 30,000 Verizon users, 500 ATT users, 300 Cingular users, 100 T-Mobile
users,
> 80 Nextel users, 10 Sprint users and 10 users of other carriers. The
> controls, and the way the users were gathered, leave too much room for
> inaccurate information.
>

When doing survey analyses total population is not a factor until sample
size approaches total population. The 0.05% of total population is a non-
issue.



Requiring 150 data point limit per carrier show CR integrity in not letting
a few responses skew the results.

"John Phillips" <[email protected]> wrote in message
news:brqOb.78457$5V2.98862@attbi_s53...

> When doing survey analyses total population is not a factor until sample
> size approaches total population. The 0.05% of total population is a non-
> issue.

You are correct. I think that people that are unfamiliar with statistics
have a hard time believing that it's the absolute sample size that
determines the accuracy, not the percentage of the total population (or user
base).

Any errors or bias in the survey are cancelled out by the methodology. If
someone claims that only disgruntled customers responded then this applies
to all carriers. If someone claims that only happy customers responded then
this applies to all carriers too. Again, they didn't ask 31,000 people who
the best carrier was, they asked them to evaluate their own carrier.

> Requiring 150 data point limit per carrier show CR integrity in not
letting a few responses skew the results.

CR is very concerned about integrity in their surveys of users. However I
think that some people fail to separate their often incomplete product
reviews from their surveys.

"Scott Stephenson" <[email protected]> wrote in message
news:[email protected]...

> The more correct way to take the survey would have 150 respondents per
> carrier, or x% of a carrier's subscriber base. The 'controls' you mention
> make my scenario more likely to occur, although not in the very sarcastic
> way I proposed it.

This would have increased the margin of error for no good reason. It's
pretty clear which carriers have the fewest responses, it's Nextel, plus
Cingular or T-Mobile (only in areas where they have reciprocal GSM sharing
agreements that began only a couple of years ago).

> Correct. But a skew in the number of respondents per carrier skews the
> results.

It does not skew the results, it increases the margin of error up to 8% (for
150 responses).

> Assuming that Verizon has a more intelligent subscriber base is pure crap.

Read more carefully. I said that the CR subscriber base has a higher
educated and higher income than the population at large. But actually I do
believe that the Verizon subscriber base is probably more intelligent as
well, simply because they don't fall for gimmicks or select a carrier only
on price or the coolest handset.

> But thanks for pointing out the biases of a CR survey in that last
> paragraph- it further invalidates the data as presented. Myjob here is
> done.

Yes, you've made yourself look like an idiot by presenting nothing to
contradict anything in the CR artticle. You can move on to other areas where
you don't know anything now.

"Steven M. Scharf" <[email protected]> wrote in message
news:[email protected]...
>
> "Scott Stephenson" <[email protected]> wrote in message
> news:[email protected]...
>
> > The more correct way to take the survey would have 150 respondents per
> > carrier, or x% of a carrier's subscriber base. The 'controls' you
mention
> > make my scenario more likely to occur, although not in the very
sarcastic
> > way I proposed it.
>
> This would have increased the margin of error for no good reason. It's
> pretty clear which carriers have the fewest responses, it's Nextel, plus
> Cingular or T-Mobile (only in areas where they have reciprocal GSM sharing
> agreements that began only a couple of years ago).

And how is this so clear?

>
> > Correct. But a skew in the number of respondents per carrier skews the
> > results.
>
> It does not skew the results, it increases the margin of error up to 8%
(for
> 150 responses).
>
> > Assuming that Verizon has a more intelligent subscriber base is pure
crap.
>
> Read more carefully. I said that the CR subscriber base has a higher
> educated and higher income than the population at large. But actually I do
> believe that the Verizon subscriber base is probably more intelligent as
> well, simply because they don't fall for gimmicks or select a carrier only
> on price or the coolest handset.

Your assumption is just that, and an uneducated one at best. The carrier
with a greater concentration of corporate, government and executive
subscribers would qualify as such, and that is not Verizon. Do some
homework and see if you can figure out which carrier that is- the answer is
no further away than a couple of clicks on the Internet.

>
> > But thanks for pointing out the biases of a CR survey in that last
> > paragraph- it further invalidates the data as presented. Myjob here is
> > done.
>
> Yes, you've made yourself look like an idiot by presenting nothing to
> contradict anything in the CR artticle. You can move on to other areas
where
> you don't know anything now.
>
>

Look in the group archive- it is very evident by the accompanying article
(which I have previously pointed out the lack of knowledge in) that they
don't have a good grasp of either the cellular market or its mechanics. Any
logical and intelligent person would agree that any survey taken where the
underlying technology and market forces are not grasped by the surveyer are
indeed crap.

"John Phillips" <[email protected]> wrote in message
news:brqOb.78457$5V2.98862@attbi_s53...

> Requiring 150 data point limit per carrier show CR integrity in not
letting
> a few responses skew the results.

One more thing, the reality is that in almost all cases the number of
responses was far higher than 150.

For example, in the San Francisco Bay Area, there were 2093 responses. Let's
assume, worst case, that there 149 Nextel and 149 T-Mobile responses thrown
out (neither made the cut), this means a minimum of 1795 responses divided
among four carriers, still a very good sample size for one metropolitan
area.

A sanity check on the CR ratings, shows them very close to other surveys
done in this area. The big exception is AT&T which has gone down over the
past year, but for a well-known reason, their botched TDMA to GSM
conversion.

There is no reason to doubt the validity of the CR survey, especially the
relative ratings. The absolute ratings may be hgiher or lower than the
general population for a variety of reasons. CR subscribers tend to be more
affluent, and hence will live in more affluent areas. But this does not mean
that the coverage is better in more affluent areas, it is often much worse
(at least in the San Francisco Bay Area, where the more affluent areas are
usually in the hills and valleys.

"Scott Stephenson" <[email protected]> wrote in message
news:[email protected]...

> > This would have increased the margin of error for no good reason. It's
> > pretty clear which carriers have the fewest responses, it's Nextel, plus
> > Cingular or T-Mobile (only in areas where they have reciprocal GSM
sharing
> > agreements that began only a couple of years ago).
>
> And how is this so clear?

Nextel failed to garner 150 subscribers in more than half the areas, and
it's a safe bet that in the other six areas that they had the fewest
responses as well. Not surprising, given their small market share.

In NYC, Cingular, which started service only two years ago, using T-Mobile's
infrastructure, did not garner 150 responses. In the SF Bay Area, T-Mobile
started service only two years ago using Cingular's infrastructure and did
not make the cut. In the SF Bay Area, Cingular is a very poor carrier, and
it was well known that T-Mobile would have the same problems, which limited
T-Mobile's ability to sign up subscribers. This extrapolates to L.A., where
T-Mobile did make the cut, but it's a safe bet that they had the second
lowest number of responses.

> Your assumption is just that, and an uneducated one at best. The carrier
> with a greater concentration of corporate, government and executive
> subscribers would qualify as such, and that is not Verizon. Do some
> homework and see if you can figure out which carrier that is- the answer
is
> no further away than a couple of clicks on the Internet.

I already know the answer to that, it's AT&T. For now. But these entities
stuck with AT&T because they didn't want to give up their numbers; they are
increasingly frustrated with the deteriorating AT&T network. That's all
changed now with number portability. What may save AT&T is the expected
merger with Cingular, and this will also help Cingular in areas where
Cingular lacks any 800 Mhz spectrum.

> Look in the group archive- it is very evident by the accompanying article
> (which I have previously pointed out the lack of knowledge in) that they
> don't have a good grasp of either the cellular market or its mechanics.

The part of the article we are talking about is a survey that is unrelated
to their evaluation of the product.
But in fact, this article demonstrated a lot of knowledge about the state of
wireless. I was surprised to see them include the sidebar "Trouble in the
GSM Network," which was right on the money. They had several warnings about
Nextel's geographic limitations. I wish they had included something about
the importance of having AMPS capability for users that venture outside of
urban and suburban areas.

Steve

http://nycell.com New York City Cellular Carrier Comparison
http://sfbacell.com San Francisco Bay Area Cellular Carrier Comparison
http://socalcell.com Southern California Cellular Carrier Comparison
http://earthroam.com International Roaming

"John Richards" <[email protected]> wrote in message
news:[email protected]...
> > "The survey was done by sending more than 100,000 e-mails
> > to online subscribers of Consumers Reports, with more than
> > 31,000 responding."
> >
> > http://tinyurl.com/34oer
> >
> > That hardly sounds like even at attempt at scientific
> > neutrality.
>
> With that large a sample size, you can rest assured that the results do a
good
> job of presenting the opinions of the average Consumer Reports reader.
> The question then becomes, how well does the average Consumer Reports
reader
> represent the average cell phone user? I'm sure not many teenie-boppers
or
> high school dropouts subscribe to Consumer Reports, so the opinions are
probably
> biased somewhat toward middle age, upper middle class,well-educated
citizens --
> all in all not a bad group to listen to.

This survey was exceptionally well designed, and any bias is cancelled out
by the methodology. The absolute numbers may be slightly different than the
population at large, but the relative ratings of carriers would not be.

This was NOT a survey where they 31,000 people responded to the question
"who's the best wireless carrier?" The people objecting to the survey
results are simply not looking at the facts, or don't understand statistics.

This is 31,000 people rating their own carrier. Unless someone is dumb
enough to believe that subscribers of one carrier are more likely to lie,
than subscribers of another carrier, I've seen no coherent argument against
the CR methodology. The fact that CR declined to rate any carrier in an area
if they got less than 150 responses, says a lot about their dedication to
accuracy.