Building on E.F. Schumacher’s famous saying, we need to find out what people are doing – and then understand why – before helping them to do it better. In other words, we need to “get behind the what”! At Eco, we’ve adopted several approaches that we use to maximize our understanding of barriers:

Start by asking “what” is happening, but then focus on understanding “why do people do what they do?”

Much barrier analysis that we’ve seen stops at the “what”. But designing sustainable energy efficiency and renewable energy initiatives requires a deep understanding of why decisions are being made. Many things that people do might seem crazy at first glance! However, there are often very good reasons why people behave in the way that they do.

For example, in many parts of Eastern Europe, including Belarus and Russia, we’ve seen people installing their own electric boilers while they are still connected to much lower cost district heating systems. This behaviour seems odd, but people are actually ensuring their own hot water supply when the community system is shut off for a month or two of repairs that happens every summer. In some countries, where district heating systems were particularly unreliable, we’ve seen other creative solutions! (see photo)

Map the market

To help identify root (and interrelated) causes, we recommend using mapping methodology that provides a consistent way of examining the situation and pinpointing the key barriers. Market mapping helps us to understand how the market currently works by mapping flows of resources or money. Then, we can better identify where to intervene.

For a project in Romania on fuel poverty, Eco mapped the flows of money, information and resources to understand how households made decisions when doing self-upgrades (i.e., DIY of low quality/non energy efficient home improvements). This analysis gave insight into the impacts of subsidies, potential intervention points for savings and ultimately the barriers to addressing fuel poverty at the national level.

Eco used a similar systematic approach to verify the market barriers to private sector investments in grid connected and captive power renewable energy in Kenya. A problem tree helped visualize the cause-effect hierarchy between market barriers. Above the “starting” mid-level key problem (“low level of renewable power production”) are the effects (such as “over-dependence on traditional sources of energy” and ultimately “high GHG emissions”) and below this are the many causes. Visually, this problem tree highlights the fact that a key barrier (problem) is the high relative cost of renewable power although this has many interrelated causes.

Facilitate the analysis

People at the local level have unique insight into what could help their situation. However, we’ve also found that people closest to the problem frequently can’t clearly see the barriers themselves! Often they need support to conduct barrier analysis and then design removal strategies. We’ve found that the mapping tools mentioned above are particularly useful as inputs to facilitated stakeholder discussions of barriers.

Understanding market barriers is fundamental to designing energy efficiency and renewable energy projects. Only by digging deep, mapping the market, listening to those who face the problem every day and continuing to ask “why” can we design a project that will be sustainable.

Let us know what approaches you’ve used to analyze barriers in your projects. Or if you’d like to find out more about how you can map your market, or better understand your market’s barriers, feel free to contact us at Eco.

The following five tips are based on our experience with top project designers and project managers who “do more with less.”

1. Make your own cuts…to indicators.  Keep the number of indicators SMALL…and make sure that the proxies used really measure a desirable change. For example, tracking the “number of policies developed,” is not a good indicator of the success of a project to influence policy development positively.  Tracking useless indicators is a waste of time and money; you don’t have that money to spare.

2. Keep the data collection simple for what you do track.  Eco has used outcome mapping when designing a multi-country project for an EU client proposal.  A low-cost system that can be run on Excel will be used to collect monthly data from network participants and then aggregate results at the international level. The frequent monitoring makes it apparent when individual teams or sub-projects are lagging behind, there is no need to invest in new systems or software, and the standardized format makes it easy to use and to roll up the results.

3. Use information that comes to you.  Make and maintain an ‘Outcome journal’ or ‘Output journal‘  –  this is an anecdotal record of any events that relate directly or indirectly to the indicators that are being tracked. This includes press releases, photos, email enquiries, etc. This type of record is particularly useful for donor reporting and for keeping track of project beyond simply completing the activities.

4. Don’t forget to ask people what they think. Use evaluation forms for trainings, meetings and other events. This step is frequently overlooked, but a simple form with a few questions can really give useful information on how the events are received and how they can be improved.

5. Measuring impacts does not have to be expensive.  Be open to hearing about unexpected impacts, and record them.  A simplified version of Most Significant Change is easy to implement informally; just ask people when you meet them, “As a result of XXX, what is the most significant change in your life / community / organization?”  The answers can provide the team with useful information about unexpected impacts for discussion at team meetings.

While the focus on generating value for money is greatest in lean times, making reporting simple, meaningful and timely is always best even in good times!

Training Needs Assessments (or Learning Needs Assessments) are one of the tools that we at Eco use to find out what training is really needed while improving the overall effectiveness of training. These assessments could help you to improve the relevance of your capacity building activities. A needs assessment conducted during project preparation or, if your project has already started, as soon as possible will make your training more targeted and efficient, lead to improved comprehension of your trainees and often save you money (more on this later)!

While Training Needs Assessments must be customized to fit the needs of your project, Eco suggests several tips that you can use. Eco recently carried out a Training Needs Assessment as part of its work on an EU-funded project in Belarus led by a SOFRECO-KEMA consortium where we put these tips into practice. In this project a group of technical specialists and managers working on energy efficiency issues was surveyed to inform the design and prioritization of future training.

Tip #1: Understand who are your respondents

Gathering some demographic data on your potential trainees allows you to understand the context of their responses better. In the Belarus project, for example, responses differed significantly between the senior managers and specialists, with managers more interested in preparing bankable investment proposals and making presentations to potential funders. Different training approaches will now be offered to managers and energy specialists. For managers, an evening session – when senior people are more easily available – will include how to use the results of energy audits and develop business plans. For specialists, a two-day detailed technical training sessions will focus on practical assignments including conducting those audits.

In many projects, it may also be appropriate to look at gender-disaggregated results as Eco has done in Tajikistan. Training Needs Assessments can easily accommodate this need by adding a simple question on the respondent’s gender, alongside other questions such as the level of the respondent within their organization.

Unless demographic questions are very short and simple, it is generally better to include them at the end of the questionnaire. And don’t ask for personal information unless you need it. While you may like to know who responds to your survey, do you really need to? Asking for people’s names can significantly alter responses since respondents then tend to answer what they think they should know, rather than what they do know.

Tip #2: Embrace open-ended questions

While many surveys are long lists of yes/no questions and other structured (fixed response) questions, we favour those that combine both open and closed questions. By handing some control within the Learning Needs Assessment to the future trainee, you allow the respondents an opportunity to think, reflect and give their opinions. In turn, the reward is a much richer understanding of the trainees’ needs, enabling you to customize the subsequent training better.

The trick, of course, it to be able to analyze the diversity of qualitative responses! (Watch out for those interpreting your survey data:  they may have a tendency to completely ignore the qualitative results!) We begin examining the qualitative responses with an eye to creating classifications or categories of quantitative responses, e.g., social/technical, positive/negative, question/recommendation, etc. depending on the question. Only then can we transfer the qualitative responses into quantitative data for subsequent analysis, similar to the approach used by the Most Significant Change technique.

Tip #3: Don’t forget learning needs assessment in your training budget

In well-designed projects, training components should include three-parts:  (1) learning needs assessment; (2) training; and (3) impact or effectiveness assessment. Often, however, these assessments are not included in the training or project budget at all. Costs for surveys can be relatively modest, but still yield useful results if well designed. Further, there are several efficient ways of both collecting data and analysing the results. In addition to the questionnaire approach used in Belarus, we have conducted focus groups (e.g., in Romania on Energy Poverty) and have used electronic surveys for large audiences. The approach should be chosen and scaled to your situation.

Training Needs Assessment results frequently have implications on the cost-effectiveness of the subsequent training. For example, in many of our projects, these assessments help to narrow the target audience thereby increasing the efficiency – and effectiveness – of training resources. In Belarus, one respondent mentioned the need for a glossary of terms to cover words like “benchmarking” and “targeting” (where an English-Russian dictionary would not shed any light on these terms for these purposes!). Providing a simple glossary to participants at the beginning of a seminar is an easy and inexpensive way to save time while improving the comprehension of the group.

Tip #4: Capture the level of knowledge before and after training

Rating the level of knowledge on potential training subjects is extremely useful when trying to understand your audience’s needs and interests. We suggest getting respondents to rate possible training subjects starting with ratings of “this is not relevant for me” to “this is something new to me”, right up to “I have a good understanding of this” and “I already regularly use this”.

After the training, the same rating system could be used to determine the overall training effectiveness.

Training Needs Assessments offer many opportunities to improve your training initiatives. The result may be a change in focus or quantity of training provided, changes to how the target audience is engaged, or possibly the realization that no training is needed at all! Overall, such assessments reward us with a greater understanding of real training requirements and ultimately a richer training experience for all involved.

What has been your experience with Training Needs Assessments in project design or during project implementation?

As 2011 draws to a close, you may also be thinking back over the past year. Take a look at what we at Eco, the project formulation specialists, have liked in 2011 and what we’re looking forward to in the New Year.

Things we liked in 2011

Taking a closer look at fuel poverty in energy efficiency projects

With projects such as this initiative in Romania to assess and address fuel poverty, we have seen this issue getting some well-deserved attention in new EU member states. A better understanding of fuel poverty and under-heating will also help to improve emissions estimates for climate change mitigation projects in the heating sector across Eurasia. For more examples from our portfolio see heating for rural buildings in Moldova, community energy needs in Hungary and Slovakia and housing energy efficiency in Russia.

Tracking changes in the attitudes and actions of key stakeholders to capture the ‘soft’ (but lasting) impacts of energy and climate projects

Methods such as Outcome Mapping are helping to capture positive consequences in energy projects that might be overlooked by traditional monitoring. These techniques measure and quantify changes in attitudes and behaviours of key stakeholder groups, and are the basis for tracking outcomes that will last beyond the project lifetime. These techniques aren’t new, but they are finally being applied to sustainable energy projects and programs such as for the Renewable Energy and Energy Efficiency Partnership (REEEP) and the International Network on Gender and Sustainable Energy (ENERGIA).

Private sector tackling some sustainable development and climate change issues

We’ve been encouraged to see the private sector step forward on some major market creation activities, including on efficient cooking stoves (Philips in India and Lesotho) and development of fuel cells (Anglo American in South Africa).

Things we’d like to see in 2012

Practical, cost-effective approaches to measure job creation in sustainable energy and climate projects

While many sustainable energy projects now address market creation, fewer are taking a close look at job creation, particularly when measuring ex-post project impacts. In 2012, we hope to see more effort to develop these methods.

Better awareness-raising and outreach

Many projects are still making common mistakes in their awareness-raising and outreach measures. In the New Year, we hope to see more projects that conduct baseline research to know their starting point, adapt their outreach to their audiences and use a media plan to increase effectiveness.

Operationalizing the Green Climate Fund

We are encouraged by the agreement on the broad design of a global Green Climate Fund. However, there is a lot of work still to be done to operationalize it! We look forward to using our expertise to develop more strategic, workable, innovative climate change projects.

  What’s on YOUR wish list for 2012? Let us know – we’d love to hear from you!

However, these are often three words for “lost opportunities.” When done effectively, education, awareness and outreach activities can encourage significant changes in attitudes and behavior, including investment behavior. Yet they frequently appear in project proposals as an afterthought. What are the most common mistakes that we’ve seen?

Mistake #1:  They don’t know their starting point

Frequently, projects designed to raise public awareness start without knowing the current level of awareness! This is particularly true for climate change projects, where awareness is almost always assumed to be very low. However, in countries where we have identified or commissioned public opinion research, we’ve found that awareness is usually much higher than assumed.

Solution #1: Good baseline research (such as a public opinion survey) allows projects to develop an outreach strategy that is more effective, identifying levels of awareness by sub-regions and by gender.  In addition, surveys give project evaluators a baseline for measuring actual impact (i.e., changes in knowledge and awareness), not just outputs such as the number of brochures produced.

Mistake #2: They don’t know their audience

Funds to support an education and outreach strategy frequently get absorbed into the project management budget, and the project team develops a “lessons learned” document that is sent to all stakeholders.  Unfortunately, busy policy-makers, factory directors and local government officials may send these reports straight to file (or to the recycling bin) without ever reading them or learning from the hard work of a project.

Solution: A strong education/outreach consultant can identify the best types of materials and settings for communicating them. Options may range from a high-level briefing with policy-makers, to a presentation at a trade show, to an article in a newsletter for municipal managers. Consultants outside of the energy/environment area (traditional private sector marketing consultants or consultants with a social marketing background) may be able to offer excellent guidance.

Mistake #3:  They start from products and not from problems

We often see projects that come up with a standard list of products – a lessons learned report, a brochure, a coloring book and 2-3 television spots – without first thinking about the messages they want to send and how best to deliver them. For example, if most people are getting their news about energy or climate change from satellite television, then local TV spots will simply be ineffective.  If people in rural areas rely on radio for their information, newspaper articles will not be helpful. And if different languages are used in different regions of a country, a monolingual brochure could overlook a key audience.

Solution: Every project should develop a media plan – an overview of key project messages and target audiences – and only then work back to its planned outputs (written and electronic).  In addition, public opinion research (mentioned above) can also identify where key audiences get their information about energy and/or climate change, and in what language.

We’ve listed three solutions to the common mistakes that we’ve seen.  What’s worked best for you in your projects?

For more information on how good project design can support best practice, please contact us.

Unilever used evolution to develop a better nozzle for the production of detergents

Developing project strategies and formulating projects and programmes that make a real and lasting different on the ground takes specialist skills and knowledge. One of the reasons is that the challenges to creating markets for sustainable energy are truly wicked!

No – we’re not making a moral judgement, but using a term from the social sciences. As Wikipedia succinctly puts it a wicked problem

“… is a phrase originally used in social planning to describe a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize. Moreover, because of complex interdependencies, the effort to solve one aspect of a wicked problem may reveal or create other problems. “

So, how does one design a project that works under these conditions?

Owen Barder, in a recent online presentation, made a powerful argument to consider the priority of monitoring and evaluation over intentional design in development programmes. His central argument is that all complex problems (i.e., development challenges that matter) are solved by evolution and not by design. Thus, variation and selection and feedback loops are the key to success.

While we do fully agree that feedback loops are a key to success, that we certainly must learn from experience, and that we frequently need to experiment and learn from what doesn’t work as well as what does, we’re not completely convinced by the suggestion that random evolution and variation is a good and efficient way to learn when it comes to real-world wicked problems.

We have two concerns: one is that we don’t really have the luxury to experiment, since failed experiments may really mean something to living human beings, often living in the context of extreme poverty! The second concern is that evolution is in fact a REALLY inefficient way of innovating. When random variations are quick to produce and cheap to deliver (such as a computer neural network, or a nozzle for making detergent) then natural selection may be a workable approach. But most great products (and projects) are the result of careful, intentional design. There will certainly be evolution in the design process (poor products will be withdrawn or improved, poor projects will fail and be forgotten, and YES, we must have systems in place to learn from project experiences), but the main innovations come from intelligent design mixed with a bit of trial and error. Surely the optimum is good design building on lessons learnt from experience, not blind evolution!

For us, this means that the intentional design of feedback loops (high quality monitoring and evaluation – M&E, and adaptive management) is a critical factor for success! M&E isn’t just an optional extra in project strategy development, but a crucial ingredient for a successful project.