Posts Tagged ‘evaluation’

When You Don’t Need to #MeasurePR

Thursday, May 26th, 2011

No Measurement!

Being a measurement evangelist feels like really hard work sometimes. On the one hand, I haven’t been at it long enough to complain — witness the indefatigable Katie Paine and Angela Jeffrey, who’ve been toiling in the trenches for, well, a long time.

But there surely are situations where measurement is unnecessary, right?

For example, you’re, I don’t know, Walmart. Your stock is suffering, there are employee lawsuits, and one of your stores has been destroyed by a tornado. How much measurement do you need to do to know you’re media coverage is, well, tortuous?  It’s likely that no amount of proactive management is going to turn your story around — at least not meaningfully.

Or, you’re a big money center bank — yep, the titans of capitalism currently getting the lion’s share of blame for the financial crisis (some of which is just wrong.) Can’t you make an educated guess about your coverage?

Aside from my personal financial stake in getting Walmart or a big bank to hire me to help them with measurement, I’ll give you three reasons why you should not measure – and three reasons why you should.

Forget Measurement When:

  1. You cannot make a difference. Sometimes business will hand you a dirt sandwich, and you have no choice but to eat it. There’s no need to weigh the sandwich, examine the types of dirt , evaluate the sandwich-maker, etc. Just eat it and move on.
  2. You’re unwilling to do what it takes to make things better.  Often, the worst media situations are when you’re “making tough choices.”  Layoffs, facility closures, moves from one city to another, hiring more executives. The path to turning the story around leads through the organization revisiting its management decisions — deciding not to outsource, keeping the plant open and operating, renovating existing headquarters rather than pitting your incumbent city against somewhere else.  See #1, above.
  3. It’s more expensive to measure than the program your measuring.  Advanced statistics are miraculous. We absolutely can measure the specific impact of public relations/communication activity on the bottom line. We just need a lot of data to isolate our impact from everything else that influences the bottom line.  That costs money (not as much as you might think, but still,) so let’s spend wisely.

Do Measurement When:

  1. You care about whether what you’re doing is working or not. You have objectives, and hopefully, they’re specific, measurable, achievable, realistic and time-bound (S.M.A.R.T.) They have a benchmark, target and timeframe. So, if you don’t measure, how do you know whether you’re making progress?
  2. You know you need to change.  Make data-driven decisions! Your intuition is flawless, of course, but as I’ve said many times, the days of PR/Communications being able to wave a hand and say, “trust me” to the c-suite are over.  A former boss told me, “facts and data win the day,” and that’s good advice.
  3. You need numbers to share with the numbers people.  Qualitative, quantitative, no matter. There are times when the people you need demand numbers. Measure to give them what they need.  Share of voice/discussion, peer comparison of tone of mention, trends in coverage overall, message presence/absence, correlation of coverage to Web traffic. Do measurement when you need to do it!

There is one other reason to do measurement — though more accurately, it’s research we want to do, not only measurement.  It’s the right thing to do. It puts us on a firmer foundation. It informs our opinions and enhances our credibility.

What’s your view?


Lies, Damn Lies, & Stinking Loads of …

Thursday, March 10th, 2011

Courtesy CBS Interactive & Star Trek

Remember that Star Trek episode where Captain Kirk is stuck on some barren planet with a 9-foot Godzilla-like lizard, and the two of them are supposed to fight rather than their respective armies? The big lizard hisses, “I grow weary of the chase. Wait for me — I will make it quick, and painless(sssss). That’s how I’m feeling about measuring social media right now.

It would be so easy to just give in.

I’ve been pondering how to measure influence, in particular, after a spirited discussion on both Justin Goldsborough’s and Shonali Burke’s blogs. That led to a bunch of posts on how we might use the structure of measuring relationships (Hon/Grunig).   This is heady stuff for peanut-brains like me.  The high-forehead types who make their living in the academe are used to thinking in these terms, but all of this stuff is pretty new for me. I’m just some guy, trying to puzzle out how to make sense of the concepts of influence in the social age, and apply the both new and hoary theories in the process. If I have to explain this stuff, I better have some ideas.

But there’s a lot more traction in just inventing a method and telling people it’s the standard, never revealing the contents of the magic box.  From Altimeter to Syncapse, to Vitrue to Klout, we learn that more-social companies have higher revenue than less-social (correlation is NOT causation); Facebook fans of a brand buy more stuff than non-fans (but which drives which?); Facebook fans are worth $3.60 (no, $136, no…), and that the “standard for influence” has something to do with Facebook and Twitter, but we’re not sure what because the formulas are secret.

H-E-double hockey sticks! I want to fight them all!

But, jeepers, why not just join them?  I came up with an idea last year to evaluate political material — know at a glance whether an article is left-or-right wing, moderate, or a combination of both.  I cooked up how it would work (programmed like automated sentiment), selected someone to write the code and even chose a name.

But it would have been a stinking load of … crap! I wasn’t basing it on any kind of research, just my own desire to make money, preferably by selling the company quickly to someone with deeper pockets, poor analytical skills and a short attention span.  Why go to all the trouble of vetting it, ensuring it actually does what it intends? That hasn’t stopped the flow of snake oil!

The class I teach at Kent State meets Wednesday nights, and on 9 March, the estimable Chuck Hemann, SVP for Ogilvy, joined us by Skype to talk to the class. He’s SUCH a smart dude (and he’s humble, claiming that I taught HIM stuff…) What my takeaway was: There are no easy answers to the social media measurement questions, and the snake oil is still gushing in the space. It takes some primary research, some actual analytical work, to figure this out. No shortcuts, no one-size-fits-all formula.

Here, I thought I’d missed the boat and should be hawking the Oil of Genius.  It’d be a lot easier than fighting the good fight, for sure. But I’m glad I’m still on the ramparts, exalting the troops to victory.

Even if I do, occasionally, “weary of the chase.”


Metrics on Relationships May Apply to Influence

Thursday, March 3rd, 2011

Creative Commons

Influence has been on my mind for a while, and on Feb. 24, I posted a thought about using the methods recommended in a paper on measuring relationships by James Grunig and Linda Hon to apply to measuring influence.  The post of Feb. 28 looked at the first three of the six components of that relationship measurement strategy.  This one finishes off the list, but I’ll have more to write on this topic later on as this all percolates.


Commitment — The extent to which each party believes and feels that the relationship is worth spending energy to maintain and promote. Two dimensions of commitment are continuance commitment, which refers to a certain line of action, and affective commitment, which is an emotional orientation.

With no criticism intended, these last two terms are a little tough for me, so a bit of explanation. Affective commitment is the sense that the organization wants to have a relationship with me, there’s a bond between us, and I value this organization over others, so it’s my emotional perspective about that relationship. Continuance commitment if the sense that I see the organization’s actions in support of our relationship… I think.

This may fit with Twitter followers or Facebook friends. If my Twitter posse is retweeting and engaging me in discussion, I can conclude they’re interested in a relationship with me.  Their actions are the continuance commitment and my own feelings about them are the affective commitment.  This type of measurement seems like a good proxy for influence, as I can conclude that the absence of such commitment would stop influence in its tracks.

Exchange Relationship — In an exchange relationship, one party gives benefits to the other only because the other has provided benefits in the past or is expected to do so in the future.

Exchange relationships are the heart of commercial propositions. We pay someone for something and get it.  But, we could say that blog consortia could be evidence of exchange relationships – we agree to promote each other’s posts and comment on each other’s blogs in exchange with one another.  I am not sure whether the extent of that relationship is evidence of influence or commerce. {and not in a bad way, mind…}

Communal Relationship — In a communal relationship, both parties provide benefits to the other because they are concerned for the welfare of the other — even when they get nothing in return. For most public relations activities, developing communal relationships with key constituencies is much more important to achieve than would be developing exchange relationships.

This one’s the stretch, in my mind – organizations have an increasingly hard time convincing stakeholders that they’re really interested in stakeholder well-being. The management-employee communal relationship comes to mind. But on an individual basis, we could say that the maturity of social media depends on creating communal relationships online.  Actual friendship.  It seems like we’d need to see a low quotient of exchange relationship if the communal quotient is high for there to be solid evidence.

I want to explore this further – and I’d like you’re help… How do your own influencers (those who influence you) align with these elements (or not?) Does this make any sense at all to you?



Measuring Influence ‘Might’ Use Relationship Metrics

Monday, February 28th, 2011


Creative Commons, by Brian Hillegas

I’ve been thinking again. Last time, I tossed out the idea that measuring influence might be gleaned from Grunig and Hon’s work on measuring relationships.  Usually, you need to get people to fill out a questionnaire to determine the quality of the relationship, but maybe looking at the public evidence is enough.  Here are three of the six elements, with comment following about the potential for adapting to qualitative influence measurement:

Control Mutuality — The degree to which parties agree on who has the rightful power to influence one another. Although some imbalance is natural, stable relationships require that organizations and publics each have some control over the other.

Think about that in context of online influence – being “Facebook friends” might imply a mutual influence, but being friends with an organization if one’s not a customer or other stakeholder wouldn’t seem to greet the same implication.

Still, the idea that an organization would change its behavior as a consequence of interaction with its stakeholders is the essence of Grunig’s Excellence Theory (two-way, symmetrical communication.) Retweeting on Twitter, and a content analysis of the @reply sequence (actual conversations) might lead to an index by topic – it could demonstrate the extent of control mutuality as a surrogate for mutual influence. The question is whether there’s enough in the stream to properly analyze.

Trust — One party’s level of confidence in and willingness to open oneself to the other party. There are three dimensions to trust: integrity: the belief that an organization is fair and just … dependability: the belief that an organization will do what it says it will do … and, competence: the belief that an organization has the ability to do what it says it will do.

This, too, could be accomplished by content analysis, substituting individual for organization. Establishing the extent of trust could also indicate the opportunity for influential behavior, which could be apparent from the stream. We’d need to define the language trusted people use, but that doesn’t seem much different from a normal content analysis.

Satisfaction — The extent to which each party feels favorably toward the other because positive expectations about the relationship are reinforced. A satisfying relationship is one in which the benefits outweigh the costs.

This one’s tough – the nature of the relationship plays in to the analysis of satisfaction. Celebrities may make general comment about loving their fans, but is that a sincere platform for mutual satisfaction? Also, if the expectations are very low (as in celebrity culture, where the connection is, um, tenuous in reality but provides a simulation of a close relationship), does that negate the influence string?  My putative 14 year-old son may get his hair in a Beiber, demand I buy Beiber music and Beiber-esq purple garments, but is that influence or a phase? Or merely effective marketing?

Next post: the remaining three elements.


Measuring Influence: 4 Learnings

Wednesday, February 2nd, 2011

Measurement isn't just bells and whistles

Measurement for its own sake is a waste of everyone’s time and money. It’s got to be in service of a strategy.

You might say that the foregoing statement is a canard; no one is beating down our doors asking us to just measure something, anything.  But there remain a feisty few, particularly on the social media side of the equation, who keep offering up horsepuckey in the guise of gold bullion.

Witness “4 Ways to Measure Social Media…,” a well-intentioned piece from last summer on Social Media Examiner. Author Nichole Kelly subheads the article with “exposure,” “engagement,” “influence” and “lead generation” — the “4 ways.”  Kelly’s on firm ground about exposure, pointing out the difficulty of a) getting good data and b) ensuring you’re counting only once, though equating reach to awareness is a colossal mistake.  Engagement,  too, is solid (if output-based), covering @replies, DMs, links clicked, comments and subscriptions. Good stuff.

Influence is listed third and lead generation fourth, showing exposure, engagement and influence as the top of the funnel leading to conversion.

The section on influence is underdone, and erroneously says tone (positive, negative, neutral) IS influence.  In fact, according to Yahoo!’s Duncan Watts, Winter Mason, and Jake Hofman, and the University of Michgan’s Etyan Bakshy, influence can’t be credibly determined from content analysis. Read all about it.

I heard Watts speak on this topic during the snowy last week of January at a meeting of the Institute for PR Commission on research, measurement and evaluation, of which I’m a member. Influence is a huge question, and Watts,’s work made me recall the somewhat hoary idea that understanding your specific audience (whether final audience or intermediary) is a lot more important than trying to calculate the exact number of impressions represented by friends of friends and retweet followers.

I pick on influence because it’s the biggest question in social media.  In fact, it’s been a big question in communication in general since the days of Lazarsfeld, Berelson and Gaudet and the two-step flow. Who are the “opinion leaders” and how do we calculate their effectiveness?

Here are four questions that hold promise when considering how to measure influence:

  1. Does the opinion leader “play” in the right sandbox for our intended audience/stakeholder?  Chris Brogan and Brian Solis have lots of followers, tribes that hang on their every tweet. Are their tribes our tribes?  They’ve got awesome scale by sheer numbers, but it’s anyone’s guess how involved they are or whether their followers in turn reach people we care about. We could get Brogan or Solis to talk about our service, product, leader or whatever, but to what end if their followers aren’t the right fit for us?
  2. Can we create a solid chain of links from the opinion leader’s actions to our desired actions?  If we’re working on building corporate reputation, retweets, Facebook “likes” and blog comments should have a relationship to opinions voiced by our final target audience. Simply passing along a leader’s statement (tweet, post, comment, etc.) shouldn’t be construed as adoption! Here’s where content analysis shows promise, especially in blogs and perhaps during Twitter chats. The opinion leader’s output should have some effect if he/she is truly influencing others. Note that this is a qualitative effort and suffers from lack of scale.
  3. Are we mistaking popularity for influence?  Celebrities routinely land atop the Twitter rankings, and there are brands on Facebook with upteen hundreds of thousands of “friends.” But having a lot of friends/followers just makes you popular. See #2 above.  We’ve long wondered about how to judge the effectiveness of influence in conventional relationships, but I don’t think many of us think the most popular student in high school was necessarily the most influential.
  4. Are we inappropriately drawing general conclusions from narrow findings?  Influence is personal and specific.  We make assumptions about readers of newspapers, TV viewers, etc., and have a body of research to back those assumptions up.  In social media, the appearance of influence may be mere output, or outtake at best. Outcomes outside of e-commerce are tough to come by, though clear objectives can solve this problem quickly.

The best measurement starts with research up front, which informs our strategy and objective-setting, followed by more research to determine effectiveness and progress toward objectives.  It’s not just tactical measurement designed to cover our butts or justify our budgets, especially when it’s trying to measure influence.


Oy, such Tsuris!

Wednesday, November 24th, 2010

I like running Communication AMMO, especially now that there’s actual REVENUE in the business. Thanks, David R.! In my Media Management course, we have to create a business, and I’ve created a Frankenstein monster. The thing is, it might be a good idea. So now I’m paranoid – should I actually make the business real, sacrifice untold thousands to try and build it, or just accept whatever grade I get and continue my march to academia? I already HAVE a business that’s (mostly) sucking cash amid hope for the future. I’m no serial entrepreneur. Feh!


Talking About PRSA, IABC, IPR on PRConversations Blog

Monday, July 12th, 2010

I’m honored (or honoured) to have written a guest post on one of the best blogs in all of PR/Communications — PRConversations — thanks to Judy Gombita, who recruited me.  The topic is my tripartite professional association affiliation — IABC, PRSA and the Institute for PR. Namely, are they valuable, necessary and a good value?  The comment stream alone is worth reading, with several luminaries weighing in (and no cursing or objects thrown so far, thankfully.) Give it a read and tell me what you think!


Internal Communications at its Best

Tuesday, July 6th, 2010

The UK’s Liam FitzPatrick wrote a post decrying the tendency of internal comms people complaining about manager communication incompetence.  FitzPatrick says: “I believe we get the internal clients we deserve.  If senior managers are used to a diet of crap communications support, that is all they’ll ever understand.”

He’s right, and he’s wrong.

The challenge always is whether to keep fighting or just give managers what they want.  FitzPatrick relates a story about a senior manager who wants “intelligence” about what employees are saying and thinking from her internal comms support.  There are a lot of things a skilled internal communicator can do to gather that intelligence, but much of the budgetary process is more output-focused than outcome-focused (echoing the same tendency elsewhere in corporate communications.)

The key for any of us is research (he said self-servingly — my practice includes research services, just sayin;.)

The research doesn’t even have to be quantitative, though tying qualitative assessment to intranet traffic, for example, can shed a lot of light on the effectiveness of our internal comms activities. We don’t have to do formal surveys, which can be very expensive and time consuming, if all we’re looking for is a snapshot to share for planning and strategy.

At Goodyear, we used an intranet poll to get just that sort of intelligence — it was a great window into what at least some employees were thinking, and it gave us a source of content, too.

But, there is no replacement for more formal measurement — even with qualification of our poll results, we still got management questions about the reach of opinion, which is a valid criticism. The old ROPE method (Research, Objective, Programming, Evaluation) still holds truth.

Meanwhile, read FitzPatrick’s piece. It’s worth reading (and commenting — no comments on his blog, so I wrote this post!)


Another IABC International Conference…

Wednesday, June 16th, 2010

I recognize that if I’m not a speaker at the big IABC soiree, I’m probably not the target audience for it. I’m not surprised, therefore, that my first blush reaction to the Toronto gathering wasn’t particularly positive.  My goal for attending this year was to meet some new people and make contact with some who I haven’t seen in a while. I hope to eventually get some business from it, but really just need to expand the network.

The programming and format are nearly identical to my first International, in 1995, also in Toronto. That one was a revelation — I was just 4 years or so into the profession, and everything was new.  Every session offered fascinating insights or enhanced skills.  I met scores of people and hung out with many, enjoying my first trip to Toronto and my first extended business trip in several years.

In 1997, L.A. was a different experience. Many of the speakers were the same as two years earlier, and in 2002 at Chicago, there were just a few sessions that really caught my eye. So I took a vacation from the big show until this year.

Things that impressed me:

Erin Dick from Pratt & Whitney — a social media case study that wasn’t from a Silicon Valley firm… Her use of blogs, Twitter, YouTube and Flickr to help support P&W’s client (the U.S.Government) on the selection of an engine for the Joint Strike Force fighter was off the charts — brilliant. And it had a fairly strong measurement component. I decided to Tweet the session instead of trying to take notes. The benefit was that I had a great summary, though my thumbs threatened to lock up from BlackBerry-itis…

William Amurgis from American Electric Power — Looking for use of social media in internal communications? Amurgis delivered. AEP’s blogs, discussion boards, employee-uploaded photos, etc., set a high standard of participation. The company’s intranet philosophy? Enhance employee productivity, reinforce corporate messages and provide a place to meet for all employees. Everything has to pass through that frame, or it doesn’t happen. And, rather than buy software solutions, AEP makes their own. Amurgis has a designer and a developer on his staff.

The UnConference — OK, it was a bit different than other UnConferences (usually low-or-no-cost, open to anyone; you had to buy the day (at least) for the IABC Conference to get in, and it wasn’t cheap) — but the method of operation was different and fun. There was no pre-set program, just a list of ideas posted on the TorontoTalks website (that a few people did discuss first), and three 5-minute “keynotes” — very informally delivered.  The three-hour session on Sunday afternoon was comprised of four 25-minute blocks of time with six possible topics (being held at six tables). We wrote on sticky notes our question or suggested topic, then stuck it on a flip chart in an empty time slot. The writer could lead the discussion, or someone else could.  I talked measurement (what a shock!) with seven other folks and it was fascinating. We didn’t solve the ROI question in full, nor did we get into other facets of communication, but it still was valuable and fun.

The thing is, the (nice) venue, formal structure and overwhelming size of the show made it hard to connect with people. Even the formal networking session (the big one held on the floor of the exhibit show) was just an hour long — not near enough time to connect. (I also didn’t attend Monday’s sessions — none particularly grabbed me. That might have inhibited my networking activities, so shame on me!)

The cost was pretty high for a new entrepreneur, not only in travel but in the conference fee. I’ll be considering very carefully before jumping on again soon. But, if I wind up as a speaker…

{FYI, I’m speaking in November at IABC’s Research and Measurement Conference in Seattle, as well as at the PRSA National conference in DC in October.  I’m also willing to come to chapter lunches, etc., and can make a deal for my PRSA/IABC fellow members!}


Crisis Analysis, SocMed Use, Get Globe/Mail Attention

Friday, June 11th, 2010

Canada’s outstanding The Globe and Mail has two stories today worth noting.  Vancouver, B.C., retailer Lululemon is using Twitter to gather intel from its customers about what sizes and colors to stock; British Petroleum gets second-guessed in its crisis communication strategy under the headline, “Lessons in Leadership Spill from BP.”

BP’s feckless communication strategy, especially demonstrated by company CEO Tony Hayward’s frequent gaffes when speaking off the cuff, deserves to be pilloried. Hayward and company were obviously led by lawyers in this regard, minimizing the potential impact of the disastrous gusher, appearing too rarely in public and pointing blame to subcontractors. Hayward’s “I’d like my life back” rang especially tone-deaf in the wake of 11 deaths and the potential for catastrophic wildlife impact (not to mention the economic peril for the gulf fishing industry.) Several communication experts get quoted in Wallace Immen’s excellent piece, including Michael Stern (Michael Stern Associates), Prof. Julian Barling (Queen’s University School of Business), and Guy Beaudin, (RHR International).

Lululemon sells athletic ware, and by all accounts does a bang-up job of it. Some of the success, according to CEO Christine Day, is due to its use of social media — Twitter and Facebook.  Reporter Marina Strauss quotes Day: “We learn more about [which items are in demand] on Facebook and social media: what are the guests really screaming for, and so we use [the feedback] to get a little bit more indication.”

Keeping an eye on its 127,000 Facebook fans and 32,000 Twitter followers gets Day and company a faster view than its store performance metrics (and offers perspectives from people who are just thinking about going to the store, rather than having bought something there — that’s an interesting view on potential demand, the pipeline, some call it.)

The social media use has two purposes, according to the article — to gather information, and to drive traffic to the company website. When we’re looking for ways to measure the effectiveness of social media, website traffic is more often cited than the research value, which is a pity.  Going back to the ROPE method of communication planning (Research, Objectives, Programming, Evaluation), you don’t have anything without the research.

If social media served no other purpose than market intelligence, it’d still be worth the investment, no?

{P.s., my Canadian sojourn is nearly complete – back to a more regular schedule next week.)