Case Study:

A RESEARCH project by Marketry into measuring the soft outcomes of schemes with young people has won a top award

Marketry, a specialist public sector research and evaluation consultancy, was commissioned by Connexions South London to find out how to measure the real, but hard to quantify, benefits of its projects such as improved social skills or self confidence. And the research was so intelligent and thorough it won the Research by an Organisation category in the National Career Awards 2006, hosted by the Institute of Careers and Guidance.

Judy Wilson, founder and managing director of Marketry said: "I am delighted to have won this award. Just because soft outcomes are hard to measure doesn’t mean we shouldn’t try. There are a lot of professionals working hard to make a difference to young people’s lives and finding a way to measure their success would justify the expense and validate their efforts as well as identifying which projects work best"

The research was funded by the European Social Fund which is running a £4 million project with Connexions South London, expected to benefit more than 17,000 13 to 19-year-olds with advice, support and personal development services.

When Marketry took on the project, the researchers looked first at defining soft outcomes and found they included social skills and organisational abilities as well as logical steps towards "hard outcomes" such as getting somewhere to live before starting a qualification.

They found that, with young people, soft outcomes could be confused with other maturing effects and changes in people’s lives so that, sometimes, what appeared to be a result of a project may have happened anyway.

And they warned that soft outcomes are not always permanent as many people go through peaks and troughs in life.

When they looked at "providers" of training or activities, they found some did not have systems and others did not collect data which made it harder to quantify their success.

Jeremy Thorn, Development Manager at South London Connexions said "the project has been very useful in analysing what providers do. Marketry carried out this project with great rigour. Everything was looked at with sensitivity and a creative approach."

Client:

Connexions South London

Jeremy Thorn
Development Manager

 

Case Study:

Evidence of impact provides a successful kick-start for Birmingham and Solihull IAG Network

An impact assessment study undertaken by Marketry has been a huge success in kick-starting a client evaluation process to provide accurate data for the Learning Skills Council and to encourage good practice within the Information, Advice and Guidance Network partnership in Birmingham and Solihull.

The main objectives of this impact study were to determine how much the services provided by the IAG Network Partnership were known to their clients, to measure in detail what steps IAG clients had taken and the outcomes they had achieved, to determine whether the IAG service met their expectations and to identify significant trends and issues.

Other issues Marketry explored for the IAG network were why some clients showed a marked preference for some services over others, what clients felt the benefits of IAG were, whether they had taken any actions following their advice and if not, why not and potential barriers to the use of IAG services.

“We chose to work with Marketry after a strong recommendation from another IAG Partnership”, explained Suzanne. “Marketry understood our business and sector so I didn’t have to explain everything to the last detail to them. This meant that I didn’t have to waste valuable time going over what sort of data we needed to obtain”.

The project was very important to bringing an external and critical view of how the Learning Skills Council were running the IAG contract and the main thrust of the study was a self completion questionnaire which was returned by 260 people. To add more depth to the study, Marketry also undertook 40 qualitative telephone questionnaires. The data was presented in the form of a detailed report to Suzanne Morey, who was very impressed with the results.

“The findings of the project confirmed much of what we thought we knew but we didn’t have the evidence to quantify”, continued Suzanne. “All the staff at Marketry were very approachable and they delivered a very professional service. Critically, all the work they undertook was delivered on time”.

Since the project was undertaken the research findings have been presented to the Steering Group which oversees the IAG programme and Birmingham and Solihull IAG partnership have worked extensively on the results with delivery providers and local partners. The study has also been useful in showing that IAG users first found out about the service from a variety of sources, with the main ones being referrals from friends and passing by premises where the service is delivered. Advertising has also played a large part in getting IAG known in the area.

The trends from this research set the scene for developing services under the nextstep branding – information and advice for adults with particular emphasis on individuals qualified below NVQ 2.

Client:
Birmingham and Solihull IAG Network

Suzanne Morey
Nextstep Co-ordinator

Case Study:

Providing evidence to gain additional funding

During their 2002 – 2004 ESF Co-Financing Programme, around £700,000 of European Social Fund cash was ploughed into advice and guidance projects supporting South London’s most disadvantaged youngsters. Connexions South London were responsible for managing the funded projects – and making sure those who most needed help were benefiting from them. They asked Marketry to conduct an independent evaluation to measure these benefits and establish how well their goals, the ESF objectives and the needs of young people were actually being met.

The task was a difficult one as Janet Cox, ESF Programme Manager commented, “the resulting ‘soft outcomes’ of advice and guidance services are notoriously difficult to measure as the perceived benefit can vary wildly from one individual to another.” Marketry’s two stage approach – two separate tranches of qualitative, in-depth interviews with 11 scheme providers and 93 young people – received praise from all parties and, importantly, succeeded in producing the hard evidence required by Connexions South London funders.

Marketry’s 360° evaluation report encompassed a wide range of performance measures, including satisfaction levels, progression routes and involvement; as well as partnership working, capacity building and best practice at a project level. It found that overall the projects were connecting well with their targets and clearly influencing the lives of many young participants.

By providing evidence of how ESF money was helping individual projects to create a significant and positive impact, and increase their delivery capacity, Marketry research played a vital part in helping Connexions South London negotiate a further £4 million of funding to help many more young people in the area to turn their lives around.

Client:
Connexions South London

Janet Cox
ESF Programme Manager

Case Study:

Market assessment and price comparison

Study Group International Limited (SGIL) helps over 35,000 international students each year with education and training, helping students from over 120 countries. When SGIL decided it needed to determine information about pay rates for UK host families, it tasked market research company, Marketry with gathering information about costs to ensure SGIL were charging a fair and competitive price to their customers.

It was important that from Marketry’s research SGIL could obtain an independent review of host family rates. SGIL wanted to use the research to determine whether their pricing was out of date and establish the differences in prices for different types of student accommodation, in different areas, also looking at seasonal fluctuations. They also wanted to benchmark their prices against other organisations supporting international students.

Marketry undertook extensive telephone research to extract the required information from schools and colleges in five designated areas in the UK. By being responsible and handling the data gathering sensitively, Marketry were able to gain the essential details.

“Marketry absolutely rose to the challenge and their extensive research delivered the necessary competitor information to enable us to place ourselves accurately in the market,” said Andrew Mills, Finance Director at SGIL.

“We were very impressed with the speed and professionalism of Marketry’s response to our needs. The staff took a very detailed brief, understood our needs and made sure they delivered exactly what was expected.”

Client:
Study Group International Limited (SGIL)

Andrew Mills
Finance Director

Case Study:

Rethinking attitudes to mental health

A statue of Winston Churchill in a strait jacket shocked the nation earlier this year.

But it made people rethink their preconceptions about mental health – and that’s just what the organisation Rethink wanted to achieve.

And when the mental health organisation wanted to find out if its campaign had been a success, they turned to Marketry because, in the words of its senior policy officer, “of all the companies that submitted a tender, Marketry had the most informed and impressive knowledge of social research”.

Marketry and Rethink developed a two-stage questionnaire in partnership for people in Norwich, where the pilot campaign was held, to gauge their attitudes to mental illness and awareness of Rethink.

Marketry found that, after the campaign, people felt they understood more about mental illness and were more positive towards people affected by it.

What had the strongest impact was the statue of Winston Churchill which stirred up controversy and was noticed by 80 per cent of the respondents.

Only 10 per cent of people Marketry surveyed felt the statue was unacceptable, showing that controversy can be a highly effective way to generate media coverage and change attitudes.

And Rethink were particularly pleased to see that Marketry’s evaluation identified areas which needed further investigation.

Adele Greaves, senior research and policy officer, said: “This extra analysis was fantastic. Marketry’s work on this campaign had such an impact that it is being used to form the basis of another similar project which we are planning for Northern Ireland”.

Client:

Rethink

Adele Greaves

Senior Research and Policy Officer

Case Study:

Measuring marketing effectiveness at City and Inner London North Training and Enterprise Council

When CILNTEC decided to reassess its marketing function, Marketry won the tender.  Marketry did extensive research in the work of the department and benchmarked its performance against other similar organisations.

‘The impact was total,’ said Peter Holmes, Deputy Managing Director. ‘A series of recommendations was made, all of which we accepted, and the result was we revolutionised the work of our marketing department.  I can’t speak highly enough of what Marketry did.  The marketing we were doing hadn’t been targeted sufficiently to be effective, now it has become effective again and has a high reputation in and outside of the TEC.  So much so that we are beginning to win business from outside.’

Marketry pitched against some very large companies and charged a fraction of the cost, and we got quality we wouldn’t have got from many larger firms.  They showed a good understanding of what government funded organisations really need.

Client:
Peter Holmes
Director

Case Study:

Assessing a new scheme at Suffolk Careers.

Marketry won a bid to provide an independent assessment of a government initiative for Suffolk Careers. The careers service needed to gauge the effectiveness of sending information about GCSE and career choices directly to all parents of pupils in school years nine and ten.  Marketry designed a unique study to show the before and after impact of the information on parents, and talk to the school about their work with the careers service.

The study showed that the information Suffolk Careers sent to parents had prompted a discussion between the parents and their children, and allowed the service to quantify in how many instances this had taken place.  They provided vital information for demonstrating the impact of the initiative. Hazel Gooch, Director at the service said: ‘The study gave us impartial feedback about what we were doing which was extremely useful.  As an unexpected bonus, we found that a lot of our work was being appreciated by careers teachers.

Marketry is a pleasure to work with – a thoroughly professional organisation.  I was incredibly impressed with the efficiency of the organisation from start to finish.  We were absolutely clear about the time scale and time after time, Marketry came up with the goods.’

Client:
Hazel Gooch
Director, Suffolk Careers

Case Study:

Introducing charges and estimating take up of a local service for adults.

In the light of careers service reorganisation, CILNTEC wanted to see if adults in its area would pay for adult careers guidance – and how much.  Marketry successfully tendered to undertake the study, and using Acorn profiling, came up with a profile of likely users and the number of these living in the local areas.  Qualitative research determined their demands and what they were prepared to pay for different services. 

CILNTEC was impressed with Marketry’s efficiency and flexibility in a highly pressurised and rapidly changing area.  Education Manager Celia Rutt said: ‘Marketry is a responsive and flexible company with the ability to adapt easily to change.  I was impressed by the way the company liaised with us and dealt with changing circumstances.

Marketry won the tender because they offered value for money – the report was read avidly by everyone involved.’

Client:
Celia Rutt
Education Manager, CILNTEC

Case Study:

Estimating the social and economic impact

For Hampshire Training and Enterprise Council, Marketry produced a report which has provoked debate within Government circles.  The Training and Enterprise Council had opened its Guidelines Information and Advice shop and Mobile a year earlier and wanted feedback on its effectiveness in the local community. 

Marketry designed a study to measure what customers gained personally, how their situation had changed and to look at the economic impact of Guidelines.  As a result of the study, the life of Guidelines was extended beyond the two years originally planned.

The measure of economic impact was one of the first undertaken for this sort of operation in the country. Discussion at Government level and within the Training and Enterprise Council will enable the method of analysis to be refined for future applications.

Leigh Henderson, Guidance Services Manager, said: ‘We value Marketry’s service very highly, they are very competent and helpful, talking through ideas for a particular piece of work and giving feedback. The service is reasonably priced.’