Remarks
U.S. Department of State Third Annual Conference on Program Evaluation - Science, Technology, Trade Track
Washington, DC
June 8, 2010


[ Get Adobe Reader View slide presentation - Modzelewski ]
[ Get Adobe Reader View slide presentation - Mikhailova ]

MR. MODZELEWSKI: Okay, so my name is John Modzelewski, I’m with the U.S. Civilian Research and Development Foundation. For those of you who are not familiar with CRDF, as we call it, we are a congressionally�'authorized non�'profit organization headquartered in Arlington, Virginia. We have field offices in Moscow, Russia, in Kiev, Ukraine, in (inaudible), Kazakhstan, in Amman, Jordan.

Though we were originally oriented almost exclusively towards the former Soviet Union, and were created originally for that purpose, to help support this kind of community in that area of the world, we are generally oriented worldwide towards international science and technology cooperation. We operate in 30 countries around the world.

And in addition to our own activities, we serve more than 200 entities in the U.S. Government and in the non�'profit community and in industry around the world with project management services, for those who want to conduct R&D activities.

So, today I am going to briefly describe our regional experimental support centers program, which I am responsible for at CRDF. It is, in plain English, a shared usage equipment program.

So, I’m going to try to keep my presentation brief, because it’s meant basically to set the table for my colleague, Liudmila Mikhailova, who conducted a detailed formal evaluation of this program recently. So this is intended just to provide you with background for the program. So this is intended just to provide you with background for the program.

And I will briefly go over the description of the program, the goals, give you some notion of the particular applications of the program and how it might be applied to other countries, talking about our selection process and criteria, some lessons learned, and why we �'�' I think the program has succeeded, overall.

So, it’s a classic fishing pole�'style program. We provide significant analytical scientific devices. We’re talking about nuclear magnetic resonance spectrometers, electron scanning microscopes, (inaudible). These have price tags in the range of a quarter�'million, $300,000 or so, generally speaking, well beyond the range of the institutes that are applying for the equipment. However, the notion is that, having received this equipment, that the institutions will then build a shared usage center around them without additional substantial support from CRDF.

Overall, in the operative period of the program, which was 1996 to around 2003, the granting phase of it, we established 21 centers in 8 Eurasian countries: Armenia, Azerbaijan, Georgia, Moldova, Ukraine, Russia, Kazakhstan, and Uzbekistan.

The idea is that the centers are to serve three different major constituencies: the educational community, the fundamental and applied research community, and the industrial community. And, generally speaking, the way the centers are organized, is that they have a sliding scale for the analytical services that they provide.

So, for example, students or Ph.D. candidates might get free access to the equipment. Fundamental researchers will build the cost of access to the equipment into their grant applications, and get a subsidized access to the equipment. And then industrial users will pay an above�'cost fee for access to the equipment, in order to subsidize the other users. That’s not the only formula that’s used, but that’s the most prevalent under the (inaudible) program (inaudible).

So, in terms of the goals of the program, it is an FSA�'funded program, entirely by the U.S. Department of State. And, as such, one of the goals is to engage former weapons researchers, either directly at the centers that we have established, or through some of the institutions that these centers serve. It’s also meant to serve as a catalyst for science and industry.

One example of this is in Azerbaijan, where the Government of Azerbaijan requires that oil companies conduct environmental analyses before they do any kind of exploitation activities. If these oil companies had to bring in foreign scientists, or they had to send the samples abroad, it would greatly increase their cost of exploitation. We helped to establish an environmental research lab in the country, which allows companies to get the environmental assessment done very quickly and very extensively, relative to other means. And, in addition, because it’s an Azeri lab, it has the trust of the Azeri Government.

Another important goal is to fill the gap left by most grant programs, including the grant programs of CRDF. Since 1995, we have been running something known as our cooperative research programs. And these are bilateral grant programs involving scientists from the U.S. and scientists from Eurasia. They are more or less on the $60,000 to $80,000 range. This is enough to cover individual financial support, or salary for scientists, their travel to international conferences, maybe acquisition for a very modest amount of scientific equipment, such as a computer, but not the big ticket items that form the infrastructure of science.

So, the RESC program has helped to sort of fill that gap at a particularly acute moment. Right after the Soviet Union broke up, the scientific infrastructure was very, very badly degraded. And, in addition to that, a lot of linkages that existed when the Soviet Union was still one country, were immediately broken up. And scientists from Armenia, for example, could no longer access equipment that they used to be able to access in Moscow.

And, finally, another important goal of the program is to promote self�'sustainability. And we’re very proud of the fact that, since the program started in 1996, we have not had to pay a dollar in salary support. Almost all 21 of the centers are still operating. In one case the equipment broke, and they have not been able to replace it. But in most of the other centers, it’s still operating today, which is remarkable because similar equipment in U.S. laboratories long, long ago would have been actually cast out or replaced by new equipment. So they have done a remarkable job keeping it running and covering the cost (inaudible).

Now, this just gives you just a few examples of the type of applications for the centers that (inaudible) the program. We could have made this slide a lot longer. But, in view of time, I just wanted to highlight just a few of them for you.

You can see that they’re essential functions for any country: public health, law enforcement, civil engineering, environmental safety. And they have been particularly valuable to countries where these centers operate. But I also wanted you to take away from this slide that the RESC program and its model could be very highly applicable to countries which are undergoing reconstruction, or where the U.S., for example, wants to promote the economic development of a country because they provide both the means to increase and improve the quality of life of a population, but also a mechanism to attract direct foreign investment to a country.

So, just a few words about our typical selection process. This is sort of a compressed description of it. It’s a little bit more complex than this, but the first step is public announcement of the competition. And we don’t even take that step until we have written concurrence from the host government or the host region.

For CRDF, we have found getting that concurrence, getting that support, that buy�'in from the host government, is extremely important to the overall success of all of our programs. When I say, “public announcement,” that’s for transparency purposes. It’s going to be on our website, it’s going to be on the websites of, say, the regional or the national ministry of education and science. It will be announced in newspapers or journals that scientists are likely to read in that particular country or region. In some cases, we have even used local partners to put out radio advertisements or radio announcements about the program. Yes, sir?

QUESTION: You said you get agreement from governments or the region. When you say, “the region,” that would be entities that would be non�'governmental?

MR. MODZELEWSKI: In most cases, it’s the national government. The exception has been in Russia, for example. The Russian Federation, we generally have been operating at two levels, both the national government, with the ministry of education and science, but also with the regional government, for St. �'�' (inaudible), and there is one more �'�' yes. And that’s been very helpful for us, because those regions are interested in economic development, they’re interested in rebuilding the infrastructure in their area. And, very often, they’ve provided very generous cost shares of the program with subsidies.

So, then we conduct an external written review drawn on our database of approximately 40,000 scientists and engineers in the U.S. And then we automatically provide, on an anonymous basis, the results of this expert review to all of the applicants, in anticipation of site visits that we organize.

We do this for a couple of reasons. One is for transparency’s sake, so they can understand the process by which we down�'select, but we also do it so that they can prepare for the site visits themselves. By seeing what the expert reviewers thought of as the strengths and weaknesses of their particular application, they’re in a better position to defend their application when we send the site visit teams out.

Ideally, we like to send site visit teams out to all of the applicants. We have not been able to do that in every single case, but in many cases we have been able to do that. And we find it’s very useful, because, very often, the applicants that have the best written application don’t necessarily have the best center concept. A lot of times that comes out only in the site visit process, and that’s due to language barriers, cultural barriers, et cetera.

We think the site visits are particular important on a major equipment program, and we have also applied them for our higher education programs, because, unlike a typical grant, where, if it’s non�'performing after a quarter or after six months or so, you can suspend it or you can terminate it, once you provide a major piece of equipment to a foreign country, getting that equipment out is exceptionally complex and expensive. So we need to measure nine times before we cut. We really need to be sure that we’re providing that equipment to the right institute. And the site visit process helps us do that.

QUESTION: What’s the percentage of success in applicants? How many are –

MR. MODZELEWSKI: Oh, I see. That’s a good question. It varies, competition to competition. I think it could be as low as 1 in 15, and it has been as high as 1 in 5, I think. But that’s just off the top of my head. That’s a good question.

Okay. So then, of course, we make a recommendation to the program funders. If it’s going to be jointly funded by the regional government and by the national government of the country, you have to have some sort of consultative process with them, to make sure that they agree with our recommendation. And they will, at times, also send a representative along on the site visit, as well.

Although, interestingly enough, they very often defer to us, because, for political reasons, they would prefer not to be sort of (inaudible) fingerprints right on the decision itself.

And then, of course, we announce the results openly. And, once again, we provide all of the applicants written results from the site visit. So, once again, they can (inaudible) their application in the future to another funding agency. And also, they can see the transparency of the process, which is really, for any (inaudible), it’s very important that you are used to funding being a black box process. This is sort of a revelation for them.

So, in terms of what we look for, and the criteria, strong support from the host institute is absolutely crucial. Because when something goes bump in the night, as it always does, you’re going to need support from your institute director. And also, if the institute director is in a position to provide any kind of support financially at all, it’s going to be very, very helpful. So it’s one of the things we look for in the selection process.

The next is a solid sustainability plan. This means a couple of things. It means getting letters of support from potential users to demonstrate that they would actually access the equipment if it was provided. It also means seeing that the institute has actually gone to the trouble of doing sort of a revenue cost analysis, indicating a realistic appraisal of what it’s going to cost to run the equipment, and how they’re going to be able to cover those costs.

You would be surprised at how weak some of the analyses that come in are. And the ones that have the best ones, obviously, tend to be the most successful.

Next is management ability, which, actually, in terms of priority, we put on top of technical capability. Technical capability is, generally speaking, pretty high at most of the applicant institutions. But an awful lot of the time the applicants don’t have a lot of experience either running a major center or in personnel management. And we find that that is actually a much more important indicator of success.

One of the things we look for when we go on the site visits are whether one person is doing all the talking, or whether they’re sharing that responsibility with the team, what kind of dynamic is going on with the team. If it looks like they’ve got one sort of charismatic figure who is not sharing with everyone else, it’s usually not necessarily a good sign for the sustainability of that particular grantee.

So, in terms of lessons learned, similar to the criteria, it’s strong support from the host institute and strong support from the government are really, really crucial for support.

Building reserve, challenging. What this means is if you are familiar with how centers operate in this country, for example, the idea would be for every dollar you take in, in either grant funds or in services that you provide, you set aside a certain amount to cover the actual current cost of running the center, and then you also set aside a reserve, such that by the time the equipment depreciates, you’re in a position to replace that equipment.

We have found it does not really work very well in Eurasia for a couple of reasons. First, over there, there can be a tendency for sort of a Socialistic attitude on the part of the institutional management. So the institute director’s attitude is, “Hey, I’m providing overhead, I’m providing the lights, the heat, I’m providing the salary. You make anything above and beyond what it actually costs you to operate the equipment, that should go back to the institute.”

Another problem is that life is uncertain. These people grew up in the Soviet period, they understand that they can’t predict which way the economy or the politics are going to go in their country. They would rather take any additional funds and immediately throw them at, for example, equipment upgrades, spare parts, supplies, or paying out bonuses, because funds that they think they have could be here today and then gone tomorrow.

And related to this, the banking systems in these countries are also still not especially stable. (Inaudible) add to that depreciation and inflation risks, so there is a certain disincentive factor. So that’s something we have to find a way to overcome if we were going to do the program again.

The next two are really related to each other, the national context is crucial in many (inaudible) sustainability, which is you go in thinking, okay, you’re going to have some sort of sliding fee structure. Especially in central Asia we have found that the governments there say, “Actually, we don’t really want them to charge users. We will subsidize the entire use of the center.” We will keep it operating.

As far as we’re concerned, if they actually do put the equipment to good use, if they are actually still using it effectively, hands off. I mean, then it’s still serving its purpose.

In Armenia, another example, or definition of sustainability is one of the centers has essentially transformed itself into the Armenian equivalent of the FDA, and it uses the equipment we provide it to do food, alcohol, and drug safety, and also to do drug and �'�' I’m sorry, drug testing and narcotics testing, and also doping tests for athletes. For all these services they provide to the government or that they provide to companies (inaudible) country, they get a fee. And that keeps their center running.

So, in terms of why we think the program has been as successful as it has, first of all, our experience in creating centers here. CRDF has created over 50 institutions in the former Soviet Union, at both universities and research institutions, and we’ve gotten pretty good at the selection process. We work very hard on our relationships with the host government, and that always comes back, and we have problems. It’s always easy to resolve them.

We have an awful lot of experience in procurement, in shipping, in technology, in licensing for exports, and also for customs clearance, not only for our own grants, but for the hundreds of clients that we serve. And this helps us when we’re dealing with large pieces of equipment getting into the country safely.

And, finally, our strong post�'grant support. We have added on to the program many grants to provide things like equipment upgrades, to provide travel and conference support, to provide training for young scientists on how to use the equipment.

We’ve also provided management training in a group setting, so that not only did they receive the management training, but all the program grantees can share best practices with each other.

And we provided, also in a group setting, English language training to help the senior management of the centers to better integrate into the international scientific community. Thank you.

QUESTION: Can I ask one more question?

MR. MODZELEWSKI: Yes.

QUESTION: How long do the centers have to agree to be open to your audit or such?

MR. MODZELEWSKI: Generally speaking, I believe it was �'�' the initial grants were something on the order of years, I want to say. I think it’s �'�' I believe it’s three years.

QUESTION: Okay.

MR. MODZELEWSKI: But the clock ends up starting again if you give a mini�'grant to them.

QUESTION: Oh, okay.

MR. MODZELEWSKI: And, interestingly, we found that the centers have been very open with us. What I mean by that is, for example, when Liudmila went out to give her evaluation, most of those centers had no legal relationship with them any more. The equipment was theirs, (inaudible) and transferred to the institutes. And yet, when we asked for permission to actually go in and take a look at how they were doing, they were very forthcoming.

QUESTION: (Inaudible) culturally unknown to them, wouldn’t it, in their past, to have that level of inspection, or just like open your books when you didn’t have to, legally?

MR. MODZELEWSKI: I mean, from my own experience, I wouldn’t necessarily say that. Soviet accounting practices are actually pretty rigorous. And the idea of somebody coming in to take a look at the books, the difference with us is I think they are a little bit pleasantly surprised that we’re not sort of the God Squad. I mean, very often we will come in, and they will tell us about a problem, and then we will try to help them solve it. And that’s a little bit unusual for them, to get that kind of responsiveness.

Thank you. Oh, sorry.

QUESTION: Yes, I was interested in your process for selecting the type of equipment, ESM (phonetic), AMS (phonetic) �'�'

MR. MODZELEWSKI: Yes, yes.

QUESTION: �'�' and (inaudible) type. I mean, was it felt that these �'�' that this type of equipment would have the broadest application for the least amount of money? Was that the analysis?

MR. MODZELEWSKI: It’s a good question. It’s applicant�'driven. In other words, for each competition we just put, more or less, a price tag, and we say, “You tell us what type of equipment you think will be the most helpful.”

And so, for any given competition, we may get 10 or 12 different types of equipment that they’re requesting from the different applicant institutions. In some cases they look for a suite of equipment that have related applications, rather than one big device. And then our experts will evaluate which ones made the best case, in terms of how they’re going to use that equipment.

MS. MIKHAILOVA: Well, hello, everybody. I am Liudmila Mikhailova, working with John. And I did the evaluation, an impact evaluation, of the RESC program in seven countries.

I have a few hand�'outs, just to follow. But I apologize (inaudible). But we will be happy to send you the whole report or any aspects of the report.

So, I am going to cover today the results, of course, of the impact evaluation program. John covered the program goals, but I still put it in one of my slides, talking about evaluation objectives, limitations, limitation constraints that I faced, challenges that I faced during the impact evaluation, to talk about the summary of findings. And I provided you with some more detailed, tabulated results collected from each RESC �'�' and to highlight some case studies that we found absolutely impressive, several RESC operating mostly from commercial contracts. And I put a few highlights on my slides.

And, of course, lessons learned. It was a good experience, and good learning experience for me to conduct this evaluation. And I would like to carry on a lessons learned with recommendations, (inaudible) program teams and to myself and to the RESC PI’s (phonetic) in conclusion.

So, John covered the program goals. And I will not spend much of the time. But definitely the most and foremost task was to establish centers of excellence shared usage facilities in the city or in the region to advance the scientific and technical fields in selected areas of expertise, and to empower the PI’s and the centers to become catalysts for approaching industry.

And this is a very tough task to accomplish, because the culture of technology transfer and commercialization is only emerging in Eurasian countries. I just came back two days ago from far east of Russia, where we did commercialization trainings at �'�' for five universities. And the feedback was that, “These trainings helped to open our minds.”

Scientists work for innovation and technology within their box, and they don’t think about the market. And talking about addressing or approaching the market was one of the major shifts in the cognitive thinking that scientists shared with me, as a result of this program.

And, of course, one of the major tasks was to engage young scientists and retain young scientists. And, as John said, in all our �'�' or across our programs, we really encourage the engagement of former weapons scientists into civilian research and direct or indirect.

Three major objectives for the evaluation study was to explore to what extent each center met the goals of the RESC program, and the set of goals that they established for their own growth and development; to study the impact of the center’s accomplishment at four major levels: professional development, institutional, regional, and national levels; and to evaluate to what extent the centers have served as a catalyst for internationalization of knowledge and institutions.

The �'�' all these countries and all institutions, when we talk about university, a part of the Bologna Process. And the Bologna Process, under the European Union, is a big deal for emerging democracies to harmonize their higher education system. So, one of the indicators of success, as I develop the instruments and measured, was how well our RESC program contributes and helps to integrate research capabilities into university systems. In other words, how, to what extent the knowledge from the RESC centers is transferred to the academia through course work and engagement of students.

It was a mixed methodology that was selected, qualitative and quantitative. And, of course, in a way, as John said, because of the context, socio�'political and economic context of each country where RESC are operating, it was impossible to generalize and to make generalization and comparison. So I selected also a case study approach, which helped to highlight the best practices from each RESC center in my final report.

And, as a result of site visits conducted from April 2009 through October 2009, 16 RESC centers were visited in 7 countries, and the 153 scientists, young scientists, RESC PI’s, were interviewed, as well as institution administrators. And, in the case of Georgia and Azerbaijan, the president of the Academy of Sciences of Azerbaijan had a meeting with me. And in Georgia, the director of the National Science Foundation also provided feedback on the RESC program.

And everything was conducted in the Russian language, and the data were analyzed and categorized in English. Having said that, even conducting the evaluation study in the Russian language where all RESC PI’s and scientists felt comfortable to communicate, there were some constraints and limitations.

And I outlined three major challenges that I faced during the evaluation. One of them �'�' probably I put it a mistype �'�' was inconsistency of the statistical information collected, not consistency. Some RESCs were integrated into institution. So the first attempt to provide information was to reflect on the growth and development of the institution, not RESC. Some of them were providing information purely on behalf of the RESC centers, so that inconsistency forced me to come back and back again, and to seek more information. And finally, I received and collected the information I wanted.

The second and very, very important challenge was misinterpretation of American terms and terminology. It was not well understood, and though �'�' again, conducting it in Russian, we were trying to provide definitions, yet the interpretation was different.

For example, “regional development.” One of the levels to measure result was the impact on regional development. In the Caucuses, Azerbaijan, Georgia, and Armenia, once they talk about the regions, they mean three countries and the neighboring country. And in Russia, of course, “regional development” means the geographical areas they are located in. So, the central Asia talks about regional development, again, on behalf of the whole region. Why it is important to define, because when they talk about the impact on the regional level, they meant different things and different scope of things.

And financial data. The financial data requested could not be adequately compared, due to the inconsistencies in the manner in which the centers recorded and provided the data. So, what I faced, as a challenge, is �'�' the first thing, not all centers wanted to disclose the real financial situation. Then I came back, and some of them did. Absolutely open, transparent. It was due to several �'�' as they explained, due to the several circumstances in the region or at the institute. It was not a good practice to share, financial data.

So then I came back in the second round of survey, and I requested percentage. And they did. Again, the percentage was tested several times, the accuracy in information was tested several times, because some of them provided that ones �'�' the percentage of the institutional budget, not the RESC.

So finally, we have �'�' in your hand�'outs there is accurate data presented and verified from different sources, institutions, RESC, and collected here as a summary based on their true income and funding sources.

CRDF has developed, over the years, the core metrics to measure science and diplomacy programs. And for this particular study, I used several core indicators of success that allowed me to measure results at full levels. As I started saying, professional development, institutional, regional, and knowledge environment level.

So, subject�'related knowledge gains and new technical skills developed, maintenance and upgrade of CRDF equipment, purchase of new equipment is one of the indicators for sustainable development. And that was the case in your table. You could see the results of the purchase of new equipment.

Engagement of (inaudible) student and retention of young scientists and, of course, former weapons scientists. The National Science Foundation came up with a new interpretation of their �'�' in one of the works they did, new interpretation brought more light to the sustainable development as a concept.

And what the NSF offered is to consider sustainable development, not only from the traditional definitions, but also to include the importance and the contribution of international and national collaborations in the human capital building and bringing a sustainability to the institutions who �'�' that operate or are engaged in the international collaborations, or national. So I included that, too.

Publications and citation index. One RESC PI, it was in Kazakhstan shared any scientific institution should be measured, or the success of any institution should be measured by two indicators: publications and peer review and citation index. And he said we have both.

Finding from the national local governments through grants was one of the indicators to measure sustainable development in success of the RESC program’s international grants and commercial contracts.

The following three slides are the summary of the findings, and I would like to leave you and give comments on each of them.

As the study revealed, out of 17 �'�' and some of you who have hand�'outs (inaudible) �'�' so John said we built 21 centers of excellence. In this study it was 18. And one center (inaudible) in Russia was excluded from the study because they were so successful, then one year later, after they were established, they were converted into (inaudible) State University into REC –

MR. MODZELEWSKI: Research and education center.

MS. MIKHAILOVA: Research and educational center. And (inaudible) and higher education program of CRDF. So it would have been not fair to other RESC programs to keep it in this study. So, 17 out of 18 on this list.

So, 15 RESC, as John mentioned, are operating today using the equipment that CRDF provided. The RESC program contributes to advance �'�' use foreign policy objectives. And this is in their �'�' in light of our conference into �'�' I put it here two ways, but practically in three major ways.

First, it adapted the best concepts used �'�' concepts on technology transfer, technology culture, and commercialization, and studied developing this culture at their institutions.

It also builds stronger ties between U.S. and Eurasian scientists. And when they interviewed PI’s �'�' and some of them are people of older generation �'�' they said, “It’s amazing.” Especially in Russia, they said, “During the Cold War, when we had �'�' we were living behind the wall, the only people who interacted �'�' was difficult �'�' were scientists. We continued to communicate with our partners in America and in Europe and other parts of the world, despite of the Cold War and censorship in our countries.”

And, of course, this program of contributing to the objectives of the (inaudible) foreign policy builds more favorable and attractive climate for the (inaudible) economic activity and direct foreign investment. And I will talk briefly in a few seconds about the best case studies.

So, the �'�' one of the major findings was that the scientific impact of the RESC equipment on the scientific community was of great value. From a financial perspective, significant difference in funding opportunities, and considerable institutional differences in assessing national and local government grants.

The increase is in the Russian Federation of the government grants. President Medvedev announced recently a new program. Fourteen universities were selected to follow. It was not said that way, but the idea behind was to convert them into research�'based universities, like American universities. And what it means is more interaction and collaboration with U.S. partners and European partners. And CRDF just recently hosted seven universities’ provosts and technology transfer managers to work with the Russian universities to advance these fields.

All 17 RESCs placed increased emphasis at the international level. RESC spent cumulatively more than 10 million on purchasing new equipment. And you have in the table the distribution by RESCs. And, of course, you could see from 19,000 �'�' which was CRDF mini�'grant �'�' like in Armenia, up to two million, where RESC were performing �'�' and purchased the equipment in (inaudible) center in Azerbaijan environmental laboratory, purchased new equipment to double the volume of the work they performed.

(Inaudible) about 3,000 users of the �'�' shared users of the equipment. In Armenia, in one of the RESCs, 14 users came to meet with me and shared their �'�' how important the equipment was for the Armenian scientists and for Armenia and the whole region. And they said that they didn’t pay, practically, as John explained. The institution transferred money from one institution to another.

So, the center doesn’t get fee for this particular service provided and for use of the equipment. But what they benefited from, from such cooperation, is publications in peer review journal, co�'publications, and advancement of science �'�' fundamental science.

Overall, not all RESCs reported, and some of them said that it would not �'�' it was not possible to count how many tests were performed. But about 5,000 analyses �'�' 5 million �'�' and tests were performed.

And, of course, the (inaudible) analytical center is one of the leading centers among eight in the far east of Russia �'�' is taking a lead in that; 68 patterns were obtained. Is it a big number or is it a lower number? It’s arguable. Some of them had zero patterns, some of them had six. So it should be judged on a case�'to�'case basis.

In 10 of 17 RESCs, 59 percent reported commercial activities. Overall, the rest have obtained 238 commercial contracts. Two more RESCs had commercial activities: one in Ukraine, The Institute of Radiology and Oncology; and one in Kazakhstan, The Institute of Chemical Sciences.

As John mentioned, one RESC in Kazakhstan, the equipment was broken and never restored. For political reasons inside the institute, the changing of institutional leadership three times, so they are recovering. But they definitely met the program objectives by engaging young scientists, by former weapons scientist engagement, and fulfilling scientific objectives.

In Ukraine, the situation was different. They were performing commercial work and commercial contracts, and were actively engaged. There is one �'�' the only laboratory, scientific laboratory in the region. But in 2003, 5 more pharmaceutical companies purchased the same equipment in the city. And by 2007, the RESC completed their commercial work, because our equipment was of older generation, and they couldn’t compete.

The equipment, I took a picture of the equipment. There are (inaudible) students working on their dissertations using our equipment. It’s still in full use for scientific purposes. But the commercial work was ceased.

One of the most impressive findings was the connection of each scientific RESC center with the academia. There are different ways that each RESC is engaged. Some of them are located �'�' an entity of the university, and are directly engaged in the academia (inaudible) that PI’s teach or bringing students for summer and spring practicum. Others, scientific research institutions, and they are engaged through collaborations and contracts with universities.

So, as a result �'�' there is a mistype, I apologize for that (inaudible) �'�' 156 Ph.D. students defended their doctoral theses using RESC equipment across the borders. More than 600 M.S. and Ph.D. students have been engaged. And I verified this information through document analysis, because they �'�' each RESC has records who and when, where engaged through direct contracts with young scientists, for example, honorarium paid or salary paid. And the same with former Defense scientists.

Three hundred plus young scientists have been working full time or part time at �'�' in the RESC project. The retention of young scientists is still a big issue and a big problem across the borders. So there are incentives that PIs are so creative at putting into the center, but sometimes they said, “Well, they are leaving for businesses.” Like British Petroleum, for example, in Azerbaijan, would hire graduates. And, of course, it’s a good collaboration, still, with �'�' between BP and graduate students.

MR. MODZELEWSKI: Our RESC had nothing to do with –

MS. MIKHAILOVA: No, no, no –

MR. MODZELEWSKI: �'�' with the spill.

MS. MIKHAILOVA: Yes, but still, I mean, it happens.

One hundred sixty�'one former weapons scientists have been engaged through the records, and I interviewed many of them. One �'�' some RESC would say �'�' 1 former weapons scientist up to 22 in some other RESCs. Yes, please?

QUESTION: Oh, I just wondered. Is there a percentage for those former weapons scientists? Like, how many are out there, and how many are being pulled in? I mean �'�'

MS. MIKHAILOVA: It’s a very good question. I don’t have this percentage. I don’t have �'�' it was �'�' it would have been very, very difficult. We have �'�' what we have at CRDF, we have the numbers from all the programs all over the years in our system. So we could extract, per country, but we couldn’t say percentage in relation to the overall. That’s a very good question.

So, 12 of 17 RESCs reported direct contractual engagement in commercial work. And talking about institutionalization of international knowledge and internationalization of higher education institutions, 32 courses are taught by 10 RESC PI’s where the courses were either created based on the new equipment, or with the use of the findings from the new equipment granted by CRDF, or revised.

And all PI’s whom I interviewed �'�' 16, 17 �'�' proved that they integrate applied research methodologies and practicum laboratory work into the education process, which is a huge, huge deal for all Eurasian �'�' across all Eurasian countries, because if you imagine scientific research institutions were always apart. And I am from Belarus, so I grew up in this system. We’re always apart from the academia. So, integration, thanks to the RESC program, of research activities was a big shift in the development of higher education institutions.

And I put some �'�' just a few testimonials from the PI’s of �'�' in Kazakhstan. “CRDF equipment was revolutionary.” And this institute, Physics of Technology, more than 50 percent consists of young scientists and engineers, which was a very good, positive development. And they were very excited about the equipment. They said, “With this equipment, we can produce films (inaudible) high�'quality testing of samples.”

Another testimonial from Professor Kadirav (phonetic), Azerbaijan �'�' this is the seismic monitoring and geodynamic center. “Equipment helped to resolve the situation �'�' geodynamic situation in Azerbaijan, and with the GPS (inaudible) technology and its practical applications in the field.” To make this story short, this particular center works as a monitoring arm for NATO to conduct monitoring for all seismic explosion in the region using our equipment.

And one testimonial that touched my heart very much, it was (inaudible) very recognized name in the Ukrainian science who said that, “In the mid�'1990s the economic and political situation was dead�'end.” And then, in red, “For the first time, scientists could see the light at the end of the tunnel. They had a new hope that the situation would turn to better.”

And I recall the time in the 1990s when I was working in the scientific research institution myself of the Academy of Sciences in Belarus. The institutes were empty. People left. There were no salaries. They were allowed to work from home. What could you do at home without any equipment? Write articles. You cannot write articles and publish and, as our people said, if you don’t have results tested.

So, it was a very, very dead end situations for many cultures in the former Soviet Union. And the RESC program was a new hope for many of them.

And one of the conclusions, based on the cumulative number of �'�' set of indicators of success, fully sustainable RESCs, 6 RESCs, 35 percent. And this is the evidence of the commercial contracts, international grants, and engagement of young scientists and former weapons scientists.

Successful developing. These are the RESCs that don’t necessarily have high percentage or percentage at all of commercial contracts, but they have a very rigorous agenda in mind, marketing strategy, new technologies, and they are striving to find commercial partners.

And organizational change needed, three RESCs, according to the results and the data collected, they would really have to undertake new initiatives to promote change within their institutions.

And last year this conference, we talked about not return on investment, but return on engagement concept. And so, I put together �'�' summarized the findings of this program. And again, I talked about most of them, but through this program and the courses and the practicum organized by the RESC centers, more than 9,000 students in 7 countries were reached or embraced, studying their equipment, and studying the capabilities of the equipment for their research.

In the few slides I put �'�' I’m running out of time.

PARTICIPANT: Yes, you should really wrap –

MS. MIKHAILOVA: Yes, I will.

MODERATOR: (Inaudible) make sure you all have time for lunch before �'�'

MS. MIKHAILOVA: Yes.

MODERATOR: �'�' the afternoon session.

MS. MIKHAILOVA: I am finishing, then. And just one of the evaluations that I did is the composition of the �'�' of each RESC, in terms of budget funding and budget sources. And so I put the average number, which covers each of the four financial sources for RESCs. Commercial funding (inaudible) is 95 in Azerbaijan. But this is an average illustrated.

The largest is increasing government funding in Uzbekistan, Kazakhstan, Russia, and the Ukraine. International funding is the largest in �'�' I will put international �'�' in Georgia. They don’t have any funding from the institutions, almost, and they develop absolutely impressive international collaborations, 95 percent in some of the cases.

And commercial activities also illustrates Armenia, Azerbaijan is the lead, with 95, 85, and 70 percent. And then the smaller portions goes to Russia. And Georgia is still striving to find.

Well, I put some examples, but we are running out of time. Just the best RESC centers, environmental, physics, and chemistry centers (inaudible) 95 percent of commercial contracts, 100 percent 2 years ago, and, because of the economic situation �'�' BP stopped some of the work �'�' they did
it �'�'

MR. MODZELEWSKI: Let me just quickly note on this particular center they did something remarkable for the region, which is they did customer satisfaction surveys. And customer satisfaction initially was not very high. And, as a result of changes they made to address that issue, they raised their customer satisfaction to almost 100 percent.

MS. MIKHAILOVA: And it’s –

QUESTION: And where was that (inaudible)?

MR. MODZELEWSKI: This was in Azerbaijan.

MS. MIKHAILOVA: Azerbaijan. And what’s also remarkable, second from the bottom bullet point, “Since 2005, the center has been participating in worldwide analytical laboratory program.” What it means, they test samples and they sent it for international review and compete. And they said that they started, when the center was open, from the level of 32 percentage of accuracy. And now they reached 92.

Still we have the goal to reach 100, but it shows also that, within these years, the center developed their expertise and is recognized internationally, not only at the national level.

QUESTION: (Inaudible) other core indicators. I think this is the worst center, in terms of publications and patents. And so, they were doing something a little bit different than everybody else –

MS. MIKHAILOVA: You are absolutely right. I double�'checked with the publications. They have two.

QUESTION: Right.

MS. MIKHAILOVA: They have two. And, of course, number of publications locally �'�'

QUESTION: Right.

MS. MIKHAILOVA: And the director said, “You know, every time we are sending an article, we get a no answer saying that the problems we describe are regionally important.” So that was the answer. And they don’t have patents, but they have certification.

MR. MODZELEWSKI: So there is a balance in every center –

QUESTION: Yes.

MR. MODZELEWSKI: �'�' between how much they do in terms of analytical services and how much research they’re conducting. And this one is very heavily oriented towards analytical services.

QUESTION: Right.

MS. MIKHAILOVA: Yes.

MR. MODZELEWSKI: It’s been an enormous success and benefit to Azerbaijan, because of that.

I agree with you. My point is that there are different –

MS. MIKHAILOVA: Absolutely.

MR. MODZELEWSKI: �'�' sort of definitions of “success” �'�'

QUESTION: Sure. That one just stood out because it was so �'�' I mean it was just really different.

MS. MIKHAILOVA: And they are different. And, of course, that’s why, in my final report, I put a case study approach throughout the whole report highlighting the accomplishment of each center, according to one or the other indicator.

So, (inaudible), as John said, is FDA �'�' Armenian FDA. And they are working now with the European Union to be a registered certified center for the European �'�' as a European branch.

And (inaudible) center I was impressed very much by the numbers in the openers. Practically they performed 4,200,000 analyses, and they said that several years ago they purchased a similar equipment of a newer generation in order to double the volume of work because of the demand in the market in the analytical services they provide in the region.

But if you look at the other tables, they don’t have international collaboration. They had one, and they are so fully operating for the regional markets, that they said, “We would welcome,” but it’s not something that they could share for the time being.

And this is (inaudible), and this is the growth of salary when RESC was started in 2005. And, of course, you see the commercial side, which is, I found, very, very impressive. And, of course, he provided �'�' the PI provided numbers. Everything was in the amount of dollars.

And, as a result also, 700�'plus publications in peer review journals. And I put a little case study of molecular structure research center. This center in Armenia is mostly focused on fundamental research. They had 370 users, and that was the center where people came and talked to me about the importance and the role of the equipment of the RESC center, in general, for the Armenian science advancement.

Are they sustainable? Good question. Probably not. They are operating under institutional budget, but they have 104 publications. And you could see in every �'�' in peer review journal, and very high citation index.

And lessons learned �'�' and this is the final �'�' so terminology, and terminology should be explained once and once again up front, and not only explained but introduced and discussed to agree upon.

Reporting mechanism. Some centers had very absolutely fascinating record systems. Some of them had good enough. But when I started digging into so how many former weapon scientists had full�'time jobs or part�'time jobs, well, this is some areas that we had to really dig and find the records from. And I don’t blame them. We are talking about 1996. So even five�'year memory, and the papers are gone.

And, of course �'�' but incentives. But it’s �'�' the European Union issued a big study on research development at higher education systems. And one of the also dilemmas they talked about is the retention of young scientists in research across the European countries.

And, of course, infrastructure building by establishing TTOs (phonetic). And this is what CRDF does, and develops now in this part and other parts of the world. And this is what our RESC centers promote as new culture and a new vision for implementation of applied research.

So, thank you very much. And I –

(Applause.)

QUESTION: (Inaudible) this evaluation influence anything? Because it’s a very favorable evaluation of a program that didn’t cost too much money and seemed to have a huge impact. But it’s over. Did the evaluation come in time to have an impact on anything? Or can it be renewed?

MR. MODZELEWSKI: Well, I don’t foresee renewal within Eurasia, because �'�'

QUESTION: Right, I’m sure you’d move on.

MR. MODZELEWSKI: But in terms of its impact, I do see �'�' as I pointed out �'�' I see a potential, a large potential, for application of this program by CRDF or by another organization in countries that are under reconstruction, or countries who are trying to promote economic development. And I think a lot of the lessons learned in this program can be applied to that sort of program.

I mean I would do certain things differently. Developing a reserve would be absolutely key. Finding a way to do that would be one of the things that, if I had to do it over again �'�' I mean, Liudmila is looking at it from the standpoint of gathering statistics. I am looking at it from the standpoint of how do you get these centers to be able to become permanently sustainable?

It’s fantastic that they’ve kept the equipment operating for so long, but it’s amazing, in certain cases, you know, the equipment really is well, well past where it should be, in terms of obsolescence, and yet they keep on doing productive work. And it’s amazing to me, but depressing in that, at some point the equipment is going to break.

PARTICIPANT: Yes.

MR. MODZELEWSKI: And at some point they’re not going to have that reserve to replace it with. And so that’s my concern, and how I would want to do it over, if �'�' again.

QUESTION: Sure. But what a great program and a great evaluation.

MS. MIKHAILOVA: Well, thank you.

QUESTION: (Inaudible.)

(Applause.)

MS. MIKHAILOVA: Thank you.

MODERATOR: Thank you very much. Thank you, everyone.