Remarks
U.S. Department of State Third Annual Conference on Program Evaluation - Democracy and Governance Track
Washington, DC
June 8, 2010


MODERATOR: (Inaudible) Senior Vice President in the McLean, Virginia, Office of Booz Allen Hamilton, where he leads the firm's organization and change capabilities in global health public sector agencies and leads the not�'for�'profit business. He has worked extensively with private sector, public sector, and NGO sector clients in the area of strategic transformation and high�'performance organizational design.

He's also co�'authored articles on the topic of strategy and implementation and developed an innovative and integrated toolkit for management techniques designed to help leaders realize new strategies and institutionalize existing strategies. He's been published in the Journal of Business Strategy and Business Horizons, appeared numerous times on ABC's World News This Morning television program, and CNBC.

Mr. Van Lee is also the recipient of the 2008 Black Engineer of the Year Award. He holds degrees with The Massachusetts Institute of Technology and MBA from the Harvard Business School.

Thank you.

I'd like to introduce also Mr. Grant McLaughlin, Principal at Booz Allen Hamilton, focuses on delivering organizational strategy capabilities with more than 18 years of experience in the areas of strategic communication, change management, marketing, public education, and with stakeholder outreach, participatory decision�'making, civic engagement, strategic management analysis, and implementation.

He holds a Bachelor of Arts Degree in Communication from Bethany College and a Master's of Arts Degree in Political Management from the George Washington University and recently earned a Certificate of Leadership from Harvard University's Kennedy School of Government.

Also we're privileged to introduce former Acting Administrator from the United States Agency for International Development, who when I was a Democracy Officer in AID would come out and evaluate our programs, Mr. John Pressley, who is also a Senior Vice President with Booz Allen Hamilton.

Mr. Pressley was the key architect of the design of the Global Development Alliance, the premier public/private partnership program of the USAID. At Booz Allen, Mr. Pressley leads the Foreign Affairs Business with management responsibility for Booz Allen's work at the U.S. Foreign Affairs Agency, the International Financial Institution, and other major donors. Mr. Pressley oversees the delivery management consulting services of other governments in over 35 countries.

I'd also like to introduce Mr. Jason Kemp, who is an associate with Booz Allen Hamilton. Jason serves as the Booz Allen Project Manager for the Department of State, U.S. Speaker and Program Evaluation, in supporting efforts at Booz Allen, focused on strategy and change management initiatives within the Department of State and USAID.

In addition to donation branding, reputation management and economic development, Jason works to support emerging market economies in Eastern Europe, Middle East, and Northern Africa.

With that, I'd like to ask for a hand and we can introduce and begin our presentations.

(Applause.)

[ Get Adobe Reader View slide presentation ]

MR. VAN LEE: Well, good morning. I'm Reggie Van Lee that he spoke about but not as interesting as my resume suggests. I apologize in advance that I'll be a disappointment compared to the rest.

What we want to do for the next 40 minutes or so is really talk about what we see as the new paradigm for diplomacy and development evaluation and the target we use to describe this is something that we have worked on over the years called mega�'communities. And soon I will define what that is and hopefully you'll have a better understanding and you can connect that to other efforts you've done and say, ah, that's what I was doing in mega�'communities. It's not a new notion but it's a new approach to problems like this.

To put it in context, this notion of collaboration and the need for the private sector, the public sector, and what we call civil society, not�'for�'profit sector to work together is not a new concept. We have quotes on the President where it says, "a government should be collaborative, collaboration, active engages Americans in the work of their government. Executive departments and agencies should use innovative tools, methods, and systems to cooperate among themselves across all levels of government and with nonprofit organizations, businesses, and individuals in the private sector."

As Secretary Clinton said, "The State Department will be firing on all cylinders to provide forward thinking, sustained diplomacy in every part of the world, applying pressure and exerting leverage, cooperating with our military partners and other agencies of government, partnering effectively with NGOs, the private sector and international organizations, using modern technologies for public outreach, empowering negotiators who can protect our interests while understanding those of our negotiating partners."

So this notion of collaboration is not new. The notion of mega�'communities started about seven years ago when one of my partners at Booz Allen approached me at one of our meetings and asked the question what is Booz Allen's point of view on globalization and how to make globalization work and how to do it in a sustainable way, and the reality was, while there were various points of views and we'd done work like that many times, there was no one firm view around it.

Booz Allen did not have its unified integrated view of how do you derive successful internationalization in that sort of way. So the question was how do we develop a point of view?

As we went and researched the topic, we noticed that there were many, many books, and we read some 75 books on globalization. We noticed that there was no real book that defined a how�'to, of how to effectively drive to sustainable globalization, and so we decided let's do some research and perhaps write a book on the topic.

It took us five years of research and writing. We interviewed over 100 CEOs, heads of state and heads of NGOs around the world. I'm proud to stay that President Clinton was one of the people we interviewed for the book, as well, and some 5,000 pages later, we discovered that this sort of collaboration across the three sectors, the private sector, the public sector, and the not�'for�'profit sector, was required in order to drive to a solution that was embraced by all the people that touched that issue and that could be implemented by all the people that touched that issue.

What we found was it wasn't just an issue of globalization but any critical problem, any multifaceted problem, be it a global problem or local problem, health problems, defense, energy, all those problems really required some sort of collaboration across the various sectors in order to drive to a solution that would be sustainable and so the working title initially for the book was A Leader's Guide to Sustainable Globalization which is a long and not very interesting title and what we figured out was really you need to create a mega�'community around these issues, so that all the various communities that connect to the issue are engaged in the right sort of way.

So at that level, the topic was quite something, but what we found from our interviews is how you make that happen is actually not as simple. Philosophically, people get it, but how do you really drive to a sustainable approach to these problems that embodies this thing around mega�'communities?

So first we started with a definition. A mega�'community is a collaborative socioeconomic environment in business, government, and civil society working together and they interact according to their common interests while maintaining their unique priorities. There's a lot of words in there.

The first thing is it's not an organization, so you can't create a mega�'community and incorporate it as an organization. It really is a network of people and an environment that work together.

The other thing we note is that it really has to be trisector. If you don't have the private sector, public sector, and not�'for�'profit sector working together, you don't really have a mega�'community and there are other ways to solve that sort of problem.

We found that people have to interact according to their common interests. So it's not appropriate for me to have you to come into the room and you help me solve my problem and a lot of times that's what's happening in these sessions. Instead, it has to be your problem and my problem together being solved in order for me to have the really impetus to put the resources and the energy into solving the problem.

We talk about the fact that I may enter the room with 12 issues I'm trying to address. You may enter the room with 15 issues that you're trying to address. We may discover that three of those overlap, and it's those three that we would work together on in the community.

So my other nine issues, I'll go home and work separately; your other twelve issues, you'll go home and work separately, but on those three we can create a mega�'community that has what we call overlapping vital interests, that we really have to have an overlap of the issues we're addressing, and they have to be vital interests.

So my priority 20 and your priority 83, we can't work them together. It has to be a high enough priority so that I come to the table and we work through the problems because there's always friction in resolving these issues.

But the last part of the definition says, "While maintaining their unique priorities." So we don't ask you to come into the room and lose your identity. It's not one of these kumbaya solutions where, for the social good, we're going to get together and make you have to leave your issues outside the door. No. You have to maintain your unique priorities yourself, so that you keep your identity, and we find oftentimes when these collaborations don't work, it's because people act out of self�'interests and if you don't play to the self�'interests, you don't get sustainable solutions.

So mega�'communities accept people's self�'interests and looks for where we have these overlapping vital interests.

So the requirements here, tri�'sector engagement, you have to all three sectors engaged, you have to have overlapping vital interests, they have to both be overlapping and vital. There has to be a convergence.

We talk about you're not here to maximize your position. You really have to optimize your position relative to others. You have to work with others. So this notion of convergence and adaptability are really quite important.

Then the last and most important thing, I would argue, is people go to conferences all the time. They meet other folks that are working on issues they're working on. They have good collaboration. They exchange business cards and then you go back to work and back to your normal day and never a chance to really interact with those people until the next conference you go to, and at some point you get tired of going to conferences where it feels like Groundhog Day. You start over each time.

Well, mega�'communities create a structure that allows people to connect through the network using information technology, setting up working groups, measuring milestones, setting goals, setting agendas, and keeps the process going, so that over time you resolve issues and you can pick up a new set of issues, as well.

So what we found is this structure or this infrastructure is what was missing from many of these collaborations, and this infrastructure can't be owned by any one of the players. It has to be owned by the network. So it has to be an independent third party. Oftentimes you'll create a 501(c)(3) that is the mega�'community that's driving this work on an ongoing basis.

So this is what we have seen as far as the creation of a paradigm that allows you to really pick up a tough issue, an issue that really requires deep collaboration across people that normally don't talk to each other or people that have very different objectives and very different views and how do you create that group, how do you task that group, how do you maintain that group until you actually start to make progress against the various issues?

Now my colleague Grant McLaughlin is going to talk about some real examples of where we have done this. We have actually not just written a book but we've actually done this in a number of ways with real success and Grant will talk about that.

MR. McLAUGHLIN: Thanks, Reggie. I have two examples to highlight today.

Jason Kemp is going to talk to you about putting this all together and walking through what we've defined as five phases, but going through we've learned that there are five phases to this program. So a couple of the examples started out may not have hit on all five cylinders.

So the first one up here, HIV/AIDS in India, this started out as a partnership with the Global Business Coalition. They believe that there is �'�' perceive this as a challenge in India, that 40 million cases, HIV/AIDS, only 10 percent knew they were infected. This has caused challenges in economics, social, big problems looming in the future. So they really want to get a handle around this.

So they laid the groundwork for those that were interested in this topic, so Phase 1, and then they worked with Booz Allen to convene a discussion around HIV/AIDS in India and the impacts that it would have and so Booz Allen designed a strategic simulation over a three�'day period where we basically executed a war game.

If we played this out, if we played that out, what would happen, and we brought together multitudes of people from across the country to engage in this dialogue that, as Reggie had mentioned, may not have ever been together before, had no real reason to come together, except in that laying of the groundwork phase we discovered they have some commonalities. They have some similar interests. How do we bring those all together and actually have a dialogue around that?

So as we've learned from this, from that strategic simulation and that exercise, we progressed on to one that actually has hit on all five cylinders and this is with the Alzheimer's Disease.

We found a partner in the Alzheimer's Association and did a number of interviews with something like two dozen different organizations that had a very vested interest in solving Alzheimer's Disease in our lifetime and we brought together over a two�'day period of time here in Washington over a 170 different representatives to discuss Alzheimer's Disease and we bound it by the United States just to have some scale to it, and we said how would we solve Alzheimer's Disease in our lifetime, and we brought together things like BlueCross BlueShield and CMS and FDA and NIH and technology manufacturers of undershirts that monitor people's body motions and body rhythms.

We brought together people who have never come together before and this time we included patients, early onset Alzheimer's, and their family who often are the caregiver, and we had a very large discussion around a simulation again over, I think it was, a 15�'year period of time. We played this out based on what we learned in the laying the groundwork phase and so that discussion came up with a wonderful set of action items that could then be taken to the next phase which really another group had to pick up and run with to design those action groups that Reggie had talked about, to design the motions that needed to keep moving.

We found a new emerging leader in that discussion which was the Center of Health Transformation which established another set of working groups that were able to take this forward and design a roadmap to solving Alzheimer's Disease and so that has moved into that fifth phase which is now around sustainability, where disparate groups are still moving against those same set of actions to be able to help solve this in the next 15 to 20 years. That's their stated goal.

So with that, I'll turn it over to Don.

MR. PRESSLEY: Thanks, Grant. The next example I'd like to talk about is the Palestine Investment Conference.

In the fall of 2007, Secretary of State Condi Rice at that time asked that a public/private partnership come together to focus on Palestine and help the Palestinians create jobs.

In addition to the diplomatic track, there's a strong economic track that says that the more Palestinians who have sustainable indigenous jobs, the more likely they are to advance to a truly functioning economy and so she asked the Aspen Institute to help form this partnership and to lead the efforts.

Booz Allen Hamilton was invited to be a member of that group. We met with the Palestinian Authority and Salam Fayad said I want to have the first time ever investment conference in Bethlehem and we said, oh, that sounds pretty good. It will take us about 18 months to sort that out and he said, no, no, I want this in five months. I want to have it happen in May. I want it to be including people from all over the world who will come to Bethlehem and see that there's a real investment opportunity here inside this country. I don't want to have it in London. I don't want to have it in Washington. I don't want to have it in Jordan. I want to have it in Bethlehem.

And so we said this will require a mega�'community approach. You can't just do it with the government. You can't just do it with Western investors. It's going to take the local people of Bethlehem. It's going to take the Governments of Israel and Palestine. It's going to take the private sector investors to come together and so we took the approach that Reggie and my colleagues laid out and we applied it in that very specific context.

From an evaluation standpoint, the idea that we wanted to be sure we had in place was to benchmark it, to start with an understanding of what we're trying to achieve. We're trying to get investment that creates indigenous jobs, so we actually had something we could measure, and then we went and met with all the stakeholders in Bethlehem and said what are you interested in doing and why do you want this to happen?

Everyone came together, sort of explained their unique perspectives on the way that they wanted this to work, and so we began to pull it together and our role was simply as the facilitator. The leader who emerged was the Prime Minister of Palestine, Salam Fayad. He was the one pushing for this, but every single element of that triangle had their own interests to make it work.

We didn't officially but we even touched base with the elements that you might think wouldn't want it to work, you know, the terrorist elements who said, oh, we're not so sure we want a conference to succeed and they, for their own reasons, said, yes, actually this makes sense.

We talked to the Israelis and we said you have to let people come in to this country and participate in this conference and they agreed that from their perspective they would do it.

We were hoping to get 2�'3�'400 people. 1,400 people showed up in Bethlehem and over a billion dollars was pledged in investments. This May, they just �'�' excuse me. Just last week, they had the second conference, two years later, and we were able to measure that those investments did happen and additional funds were pledged this year and that mega�'community approach, we believe, had a major impact on having this be a much more successful event than if it was just driven by some outside institution.

Thank you.

Over to you, Jason.

MR. KEMP: Thanks a lot, Don. I'm going to stand up around here so everyone can see me.

So I think Grant and Don provided some very good examples of sort of mega�'communities in action and what I would like to do is take the opportunity to tell you a little bit more detail, particularly from an evaluation perspective, and, of course, as a consulting firm, we can't help ourselves sometimes, so we have a nice graphic up here with different phases, etcetera.

So if you just allow me just a few minutes to sort of walk through what we call the mega�'community life cycle development model, and then we can go into more specifics around the sustainability piece which Grant mentioned earlier which is where the evaluation component is the strongest piece of this particular model.

So what we have here is the current version of the mega�'community life cycle development. What I mean by current version is that one of the things that's very important for us as an organization is to make sure that we learn from every mega�'community. No two mega�'communities, Reggie mentioned this earlier, are the same, and so it's important for us to take lessons learned from what we may have done in 2005�'2006 when we were in India and what we've learned with what we did with the Alzheimer's work and what we learned from Palestine and really put that into our overall life cycle development model.

I'm not going into each one of the specifics, but I do want to focus on the fifth phase here which is sustainability and sustainability, if we can circle it, next slide, right here, development, evaluation, measurement, and reporting tools, and that's really a key component of the framework and that's one of the things that has been very important to us as an organization.

It's not just stating there's a need for a mega�'community, let's convene a mega�'community. Let's look at how to make mega�'communities sustainable and so we felt that that was a particular piece that would be interesting to this audience to focus on as we talk about evaluation from a diplomatic and development perspective, and I think one of the things that we've seen from our international affairs engagements that we've participated in are a couple of lead�'in indicators that talk about how mega�'communities can be applied in this space.

So I really want this to be a back and forth conversation. So I'm just going to sort of touch on a few high points and then I'm going to sit back down so we have plenty of time for Q&A and have a good discussion around mega�'communities and evaluation, but I did want to sort of throw out, for lack of a better term, some food for thought with some scenarios based on some work that we've done in the development and diplomacy space. And a lot of that work is based on sort of these three tenets of stakeholder identification, what we call relational network analysis or social network analysis, and then social collaboration, social media, government 2.0 or just web 2.0 in general.

These are capabilities that the firm, we as a firm, have invested a lot of time and resources in to understand how to make that applicable in a development and diplomatic context, in addition to other contexts for services we provide for other clients, and one of the things I think is really interesting about the blending of these three, I would say for lack of a better term, service offerings or capabilities is that it really does speak to the evaluation point.

It speaks to what Grant and Reggie and Don were referring to which is the network, which is looking at the network, looking at convening a mega�'community, you know, establishing the parameters of the mega�'community because if you can establish the parameters, if you can map the network, if you can track awareness and desire and see who are advocates of particular issues and who are supporters of the gentleman or gentlewoman who might bring 20 issues to the plate, where the other individual might be 15, if you can track that and then maybe create an online platform that will allow for some kind of communication and, of course, that depends on a multitude of variables, particularly in the development space, but it's the blending of these three tools that really allow you to create an opportunity to look at sustainability through evaluation.

And the next slide just talks about some, what I would phrase, mega�'communities in action, but this is really based on a lot of work that we've done as a firm and we've supported a lot of work around regional trade agreements, WTO session, EU session.

One of the things that was an interesting take�'away from that work is the need for the tri�'sector engagement piece and I think one of the great things about the U.S. development model is that there is a focus on metrics measurement evaluation, but from our perspective, it seems like the mega�'communities approach would be a great way to look at supporting countries who are trying to align, you know, various Rule of Law, democracy, governance, whatever you want to phrase it, in terms of looking to make sure that they're aligned from an economic liberalization perspective, I guess, to join the WTO.

So there is things that you can use the mega�'communities for in that context and then just large�'scale multilateral diplomatic efforts. I know that sounds very consultancy and probably doesn't make that much sense, but what I'm referring to is what Don mentioned with the Palestine Investment Conference.

We think that mega�'communities is an approach to look at what are dynamic and, quite frankly, sometimes complex diplomatic situations and that have the development component to it.

And then one of the things also is looking at multilateral institutions. You know, we know that the World Bank, through the work that we've done in collaboration with the World Bank, has been a lot around social collaboration and innovation and so looking at mega�'communities through that in terms of building partnerships with various in�'country organizations, etcetera, might be a good approach.

And then, finally, in general, and I see a typo that I created there, which we are the United States, not the United State, but global engagement is a hot topic right now in not only Washington, D.C., but, you know, throughout particularly within the national security and foreign policy spaces of our government.

I think mega�'communities certainly doesn't offer a solution but mega�'communities offers a beginning of a discussion. I think our firm would not want to come up here and say to you that mega�'communities is going to solve all the challenges related around evaluation in regards to diplomacy and development.

What I think we're saying is in this era of global engagement, to take it back to what Reggie was saying earlier, with President Obama and Secretary Clinton, in this era of global engagement and the need behind it is mega�'communities a new framework to look through some of the issues that we're all facing in this room as the organizations we work for, whereas from a private sector perspective, the organizations we support, is this a framework that we can use to help really advance evaluation techniques in this space?

And so I will quit talking and sit back down. I think we can look to you all for Q&A.

MODERATOR: Thank you very much, Jason, Grant, Don, and Reggie, and now I would like to open it up for the floor for any questions. Please.

QUESTION: Francis Javez, Swedish Embassy here. Don Pressley, I mean that was impressive what you said about the Palestine conference and that was now two years later, the second conference, but you told us about $1 billion pledged. Can you not tell us how many jobs were created with those one billion?

MR. PRESSLEY: I can't off the top of my head, but I'd be happy to follow up because we have been tracking that and the Palestinian Authority has been tracking that, and it's a very large number. I just apologize, I didn't bring the information.

QUESTION: Because you said that this was, you know, something rather concrete that could be measured and this is �'�' I mean, I think we really need to get down to, you know, the nitty�'gritty here because otherwise we talk around things and it all sounds very good, but we must be able to see exactly what has been achieved with the investment.

MR. PRESSLEY: I absolutely agree with you. There must be measurable results like that for exactly what you want to try to do and it's an ongoing effort, and it's the kind of thing that sometimes is hard to go out and identify exactly how many jobs have been created and we've stepped out of the process since then and they're no longer relying on us. We sort of turned that over to the Palestinian Authority and they've been doing it themselves and they ran the second conference themselves. So that's why I don't have the exact information at my fingertips.

QUESTION: I'm from Chicago, and it seems to me that the mega development communities is a development tool. It's not an evaluation tool. You evaluate the results of implementing the mega�'communities.

So when you said the Palestinian number, the only really concrete one, the real question is what would have happened if you didn't have the conference and do the mega�'communities thing, and so then you answer the standard evaluation paradigm.

So do you have �'�' you said �'�' I mean, I thought the question was, are there concrete estimates and presumably it's not just the number of jobs created possibly linked to those particular investments but what investments would have happened without the mega�'communities, without the conference. And it's the difference between those that you're �'�' should be attributed to this activity.

So I'm trying, I'm just trying to figure out where �'�' but, first of all, tell me whether mega�'communities is a development tool, not an evaluation tool. Let's go from there.

MR. VAN LEE: Certainly, mega�'communities is a development tool, but as Jason was pointing out, when you look at the fifth phase, the sustainability phase, being able to evaluate around that mega�'community's approach is something that makes it an evaluation tool.

Jason, you might want to comment on that.

MR. KEMP: Yeah. I think one of �'�' I think it's a great question. I think one of your points was getting to, you know, what we oftentimes call absence of the intervention and I think what you're referring to is if there wasn't a particular investment conference, would these other indicators �'�' how would they have shifted, how would they have changed, and I think that it's a very interesting question and in fact I can't speak to the Palestine Investment Conference, but I can speak to some work that we're currently doing in regards to supporting evaluation efforts within the Department of State.

And one of the things that we're doing is we're looking at just that, we're looking at public diplomacy tools and we're looking at the absence of that intervention, so the absence of that public diplomacy tool, and what we're doing is we're using advanced social science techniques around social network analysis and relational network analysis to get a baseline understanding of the dynamics around a particular public diplomacy intervention.

And then we're controlling for that with various different participating groups to essentially take the intervention out and so we do demographic mapping with other particular participating groups with similar demographic backgrounds and see what leading indicators or what activities didn't happen and then use that to map it back to the absence of the intervention.

So I think you actually raise an excellent point about looking at mega�'communities and asking about it as a development tool, how do you use evaluation. But one of the things that we've seen so far in this space is looking at exactly that, which is absence of the intervention which, as I know, is all of us who are looking at evaluation and the diplomatic and development space. I know that's a critical question. So thank you.

MR. VAN LEE: I would just add two observations. One, I think it's both a development tool and evaluation tool. The premise of the mega�'communities is that you have to engage these people, otherwise certain things won't happen.

So in all of the meetings when we do the simulations, the simulations test the what�'ifs and looks at the if you intervene, if you didn't intervene. It is theoretical to some extent, but it creates the mega�'community around the premise that in the absence of doing certain things and things happening in the environment, what would you do?

My observation is we did this work at ENIL, which is the energy company in Italy, who launched this initiative to build a new plant in Brindisi, which is a region of Italy, and they talked to the local people and they thought they had all their ducks in a row and they launched the effort and five people with picket signs with the media's attention shut down the plant and it cost them $30 million.

We then came in and applied the mega�'community approach and were able to drive to success. So to some extent, we've tested and perhaps it's, you know, after the fact, when you didn't use a mega�'community versus when you did, but I think in the simulations we do, our goal is always to create the mega�'community, having looked at what�'if scenarios and what if you didn't have the interventions.

QUESTION: Thank you. My name is Katherine Thorben. I'm at the Peace Corps, and when Don and I were at USAID about 15 years ago, we started working on something called the New Partnerships Initiative which was sort of an initial effort to move in this direction.

And one of the things I liked about what you said, Reggie, is that a mega�'community approach is not always the ideal approach. Sometimes it's more effective for one sector to pursue its objectives on its own, but there are going to be issues where much more can be accomplished than any one sector could achieve on its own.

So I think it's important to keep in mind that sometimes it's the right approach. It's not a panacea, but I think that this focus on what is the value added of this approach, what would have happened had this approach not been taken, you know, that's really the critical issue to being able to substantiate that this approach is something that will give you more than would have been accomplished with each sector trying to work on its own.

MR. VAN LEE: In all fairness, oftentimes we create mega�'communities around issues that people have tried to fix by themselves and couldn't. So oftentimes it's the last effort and they say, well, let's try this and we've had little success with it, and there are cases where either the environment is unready for mega�'communities.

The issue seems ripe but the people just haven't gotten to the point that they're really willing to be objective in how they drive it. So they're still in the mode of I have a problem, you help me with my problem. They haven't gotten to realize that I've got to give up something and in giving up something I'm going to get more than I've given up. So you're exactly right.

QUESTION: Your sustainability or phasing chart, if you could go back to that just because it's easier to look at it that way.

Now, I'm assuming that you don't do those things �'�' no. The one where you had the five phases. That one. I'm assuming you don't �'�' you do some of these in tandem as opposed to one, two, three, four, five. My concern in looking at if you're doing the evaluation or developing your measurement system at the end, what if you end up measuring the wrong thing?

To me, that looks like you're doing it as an afterthought when really that whole what are we going to measure, what is the outcome kind of the begin with the end in mind kind of aspect really needs to be built in from the very beginning. Am I looking at that correctly or �'�'

MR. VAN LEE: I would argue that five sort of runs along one, two, three, and four.

QUESTION: Okay. Well, that's my question because, you know, there's a lot of times that metrics measurement, monitoring and evaluation, whatever you want to call it, is the afterthought or we forgot to do that, what are we going to do. You measure the wrong thing or it's easy to misinterpret the results. So I just want to make sure that I was looking at having that correct mindset.

MR. VAN LEE: The one observation I'd make, though, is oftentimes the initial metrics that we put in place, we discover over time were not necessarily the right metrics, and so there is an interaction even around those metrics.

Oftentimes we're measuring symptoms of the problem as opposed to the real problem, but you can't force people into that. They have to sort of discover that. So though we at home may know the real set of metrics should be X and Y, what the group is willing to adopt are A, B, and C, and over time we move them to X and Y, but it does move in parallel from the beginning.

QUESTION: Okay. Then does each group that gets involved in this, you know, in your mega�'community, are there, say, kind of an overall, you know, arching kind of global level of metrics as opposed to �'�' and, in addition to then the my group needs to measure this, then your group needs to measure that and so on and so forth?

MR. VAN LEE: It tiers up. There's like a tree or pyramid up to the ultimate goals and then the piece parts and sometimes it takes awhile to figure out what are the piece parts that drive to the ultimate goals. It really is an iterative process and you really have to suspend your disbelief to some extent, but you find ways to get people engaged.

Without the metrics and without hitting the milestones, people won't stay onboard. So we have to set the metrics upfront and some milestones and say, all right, you see in the first year we did achieve X, Y, and Z. This is why you should continue to be involved.

MR. PRESSLEY: I'd like to add to that that we also have to recognize that this is an evolving kind of thing. As Grant and Jason said, we're learning as we go and it's rare that you have the opportunity to do a full�'blown baseline evaluation study upfront in order to work from that.

I was involved with the startup of the Program of Assistance to the former Soviet Union and Larry Eagleburger told me no needs assessments. We don’t want to assess their needs. We want to just go in and help them and so then we could never figure out whether we'd done anything or not.

So that's the real world environment that you live in and I think part of the evaluation techniques that have to be adapted to that sort of real world situation is you are going to have to, you know, come up with surrogates for a very rigorous kind of evaluation methodology and the kind of opportunity that Jason was talking about is actually pretty rare, at least in the kind of work that we see.

MODERATOR: Thank you. You had a question?

QUESTION: Andy Blom. I work with U.S. Institute of Peace Grant Program, and my job is to support civil society organizations and so I wanted to ask about the sort of power and resource disparities that are in this triangle and it goes to the metric question because, you know, whose goals are being reached, whose goals are getting measured, and with the process, I mean, is there more support provided to that triangle just to get to meetings, just to build up their capacity to deal with some very powerful actors in sort of the other two parts of the triangle?

MR. VAN LEE: That's interesting. Oftentimes there's a difference between power and resources and we find that private sector has the resources but oftentimes doesn't really have the power. They think they do and the ENIL situation in Italy proved that five people with, you know, picket signs could shut down this huge endeavor.

So we recognized upfront that if we do the stakeholder analysis, we have to be very careful to understand who has the real hidden power in the situation, but they have capacity issues, and how do we build up those capacity issues, and oftentimes it's not money, it really is intellectual capital, the intellectual horsepower that we add to the picture, as well. But that is an ongoing challenge.

The real challenge is a lot of times the civil society participants that are least willing to collaborate with each other because they see each other as competition for donor dollars. So that's the thing we've had to �'�' and we talk about the Alzheimer situation, but, you know, we had a challenge with the not�'for�'profits involved in this, not so much with the pharmaceutical companies and the manufacturers of devices and that sort of thing but with those not�'for�'profits to get them to really come together and then to get them to accept the fact that they needed capacity support and to give them that capacity support.

But at the end of the day, they have enormous power. I mean, they have the voice of the people that others don't have.

MR. PRESSLEY: I think there's a trust issue there, too, and the civil society advocates are often so passionate about their stand on an issue that they're unwilling to make the compromise necessary in order to achieve what would be a better result than not getting your position approved at all.

And in Reggie's book he talks about how do you get the protagonists to understand that there's an optimal solution that would be better off than each one trying to maximize their own position and that art of compromising does involve power and resources and roles but as Reggie just said, it's often the one who's willing to keep pounding away on the issue that in the end can have a lot of influence over where is that optimal solution.

MR. KEMP: One other thing that I'd like to offer up is the addition in the Alzheimer's game that we played this simulation, the addition of patients and caregivers was a great grounding point to keep everybody on task and on focus and making sure that there's a rallying cry around that. So that was a great convener to get them there and it was also a very large component to keep them there.

MR. VAN LEE: I have a confession that a week before the Alzheimer's event, someone asked the question, have we invited an Alzheimer's patient, and we had not. So we were able to get somebody there and we had them in all the groups and it worked out fine.

But in many cases, though, the last person we think about are the people that we're there to serve the most. We have people that represent their points of view, that understand exactly how they feel, which they don’t, but that's what they profess.

So I think this notion of where's the power in the room really �'�' and it did change the dynamics of that. We started with an Alzheimer's patient speaking about his experience and had caretakers there speaking about it, as well, and so we came to real solutions as opposed to the solutions that theoreticians would put together.

QUESTION: Hi. My name's Sarah Brewer. I'm with the Evaluation Measurement Unit in the Public Diplomacy Bureau.

I guess I have a comment and maybe trying to see if there's any work that you all have done. I'm familiar with the capabilities of your SEDC and the relational network analysis work that you all do and so there seems to me to be sort of two things that are happening here.

One is, and identified by the colleague back from Chicago, around the need for sort of objective metrics around whatever project it is that these different sectors are coming together, the problem that they're trying to solve.

So whether it's the decrease in �'�' you know, the cure for Alzheimer's or the jobs in Palestine, you know, there's a set of metrics that we can objectively sort of see if we're driving to the goal, but I think what's very exciting about sort of the approach, this perhaps development approach and the capabilities of the firm is that, you know, the relational network analysis in particular would provide you all with the opportunity to go in and sort of see these power relationships and measure these power relationships between the sectors and the actors and really provide some, I think, quite useful feedback around sort of the relationships that are needed in order to, you know, achieve success or sustainability and you all can sort of identify those key points, right?

So when you're working together on another project, you can say, well, you know, we need to figure out, you know, whatever the names of the dyads would be or the triads would be, you know, and sort of really flesh this out in a way using the analytic tool that could provide, I think, more just, I guess, information about the model and how it can be applied in future projects, really, right?

So I guess the question is have you all done that? I mean, on this Alzheimer's work, you have all these wonderful sort of this data here with all these people running around and this tool and, you know, have you sort of talked to them about how they all work together and sort of tried to flesh that out because that would be very interesting, I think?

MR. VAN LEE: Yeah. As Don said, it's evolving. I mean, we are still doing our own homework at home to make sure we're pulling together learnings, but the good news is we have people from each of the pieces of engagement that do the next engagement. So there's some institutional learning that follows, but we've got to get to the point where we systematize it more and a lot more of it can be systematized than you would think.

A lot of this behavioral modeling and seeing how different groups will still behave the same sort of way, given a set of conditions, is something that we're carrying forward. So Caroline knows that we're trying to get an investment in PMO to do that, to pull those learnings and to get that more institutionalized.

QUESTION: I wanted to go back to the point that you made, Reggie, about this question of disparity of power because I think, to a certain extent, it's a bit of a red herring and it really has to do with the fact that civil society organizations frequently sell themselves short.

They underestimate the power that they have and the special expertise that they bring to the table. If you're talking about their knowledge of local needs, you talk about the trusts that they have in the community, you talk about their mobilization capacity and their technical expertise, and so I think frequently the challenge in putting together a mega�'community approach is convincing civil society that they're bringing a lot to the table and recognizing what their special expertise really is.

MR. VAN LEE: And that's one of the first things we have to do. As a matter of fact, there's more to talk about than we have time to talk about, but the most important role, I would argue, in these mega�'communities is what we call the convener, the convener role, and the convener is seldom the private sector and is many times not the public sector. It is oftentimes the civil society.

They have the power to bring people to the table and their motive is a little less suspect than anybody else bringing people to the table. So we have to convince them sometimes. They want Booz Allen to be the convener. We said no, we should not be the convener. Alzheimer's should be the convener. Diabetes should be the convener. The Veterans Coalition should be the convener and oftentimes they say we're understaffed, we're under�'resourced, we don't have the power. We have to help them understand through the stakeholder mapping the power they really have and then get them to exert their power, as well.

QUESTION: Carol Horan, Bureau of Educational and Cultural Affairs here at State.

You've used the term "simulation" for different programs that you've done and when I think of simulation, I think of like, you know, an emergency simulation where there are people pretending to be injured and bloody and whatever and obviously this is not that.

So if you could give a little more detail on what you mean by simulation, maybe in the Alzheimer's situation, for example. What exactly happened? What went on?

MR. KEMP: Sure. So as part of the convening component of this, we brought, as I mentioned, about a 170 people together in a ballroom here in Washington, D.C., and we had devised a number of steps that each group would be moving through as a collective body and that we broke up. I think there were nine different teams, maybe 10 different teams for those 170 people and there is a group that focuses in on patients and families. There's a group that focuses in on financing. There's a group that focuses in on the research or the technology or the media, right, and so some segment of the population gets together and they work in different pods.

MR. VAN LEE: We cross-fertilized those groups. So even though you may have been in the media, you may have sat on the patient team. So we did some cross-fertilization. We had experts on the team that knew that space and we had other people because the goal was to get you to understand the perspective of somebody that you aren't oftentimes. I'm sorry.

MR. KEMP: Yes. So there's a nice blend there. So if you're supposed to be representing the media, there's enough sprinkling in the discussion to represent the media as part of the group, and there was a control group, as well, that would help monitor the different components. It would play things like the White House or Congress.

It would inject different dialogues at different points and also help serve as a mechanism to have the various disparate groups talk to each other throughout the period of time because, as you're going through this, you can't necessarily solve it within one set of �'�' within the set at your table. So you need to be able to have conversations with patients or families or conversations with the government.

So during this period of time, there were multiple moves and those moves were designed over �'�' it had a time period element to each of them and those moves set advances in research, set advances in policy or changes in the landscape, whatever the scenario was happening, and then during that period in time, which was probably 45 minutes to an hour and a half, right, depending on the length of the discussion, the group has to come through three or four big sets of questions that were devised in the Laying the Groundwork session.

So in order to be able to achieve something, we've devised these three or four questions that your group needs to come up with an answer to and you're able to talk and dialogue with others in the room through that control group and then come up with some sort of answer at the end of that period of time that you're going to present back to the large discussion.

So there's a series of initiatives that come out of those things. There's an out�'brief that's probably a dozen pages long that gets to these series of initiatives that this group as a body said yes to and we're going to, you know, continue to work through different action groups. So in a nutshell that's how it plays out.

MR. VAN LEE: And afterwards we can talk more, if you want, more detail on that.

MODERATOR: That's basically the time up. We really want to thank all our presenters, Jason, Grant, Don, and Reggie, and give them a nice welcome. Thank you.

(Applause.)