Automated Customer Surveys

Technology from Database Systems Corp. lets you develop IVR survey applications using our interactive voice response IVR solutions. Surveys can be initiated by outbound phone calls or can be a response to callers. Using our PACER and WIZARD phone systems with the Smart Message Dialer and survey software, we can call your survey prospects and play a highly focused and custom greeting. We then can give your survey audience the option to take your survey or even talk with a representative, leave a voice message, hear additional information, or simply decline to participate in the survey. The survey can accept touchphone response or can record each question response for later analysis. To view more information regarding our automated phone applications, please visit our Automatic Phone Survey solution web page.

The following is an article relating to automated survey techniques and products and services in our business.

INTRODUCTION

1.    Why Conduct a Survey?

Librarians, like any other managers of organisations serving the public, need objective feedback from the users (and sometimes the non-users) of their library.  Such survey information helps you plan ahead, helps you monitor the impact of various changes you might have made, gives you users' insights into the way the library works, gives objective feedback on performance etc. and may even be part of the accountability measures that have been written into local government plans.

2.    What is the Survey’s Purpose?

The first thing you need to do is clarify exactly why the survey is being carried out.  You need to decide what information you want to collect, who from and how it will be used.  Once you have done this the design of the survey will flow logically.

You may need to contract some parts of the survey to outside contractors or even everything to an outside agency.  Contracting out survey work is desirable if you do not have the skills, staff, or time to do the survey in-house.  Contracting out survey work is useful if you need to demonstrate that the results are objective and unbiased by management.

On the other hand if you do the survey in-house you will develop valuable survey skills which you can use again.  Also, in-house surveys generally cost less than ones that are contracted out.  Even if you think you will contract out the survey you still need to have a good idea of what you want from it.  It is always beneficial to carry out your own preliminary research (by talking to a number of people, including potential respondents) even if you do intend to brief a contractor.  These guidelines will give you some basic knowledge about what to expect from a survey.

3.    Who to Survey?

You need to be able to describe who it is you want to survey, often called the target group.  Your target group could be all members of the general public in your library's catchment area or all adults in that region or all frequent users (say at least monthly) of the library or non-users of the library etc.

Now you have to decide whether you want to interview all members of your target group (a census) or a random sample of them.  If your target group is small it may be possible to survey everyone, however if your target group is large you may need to survey a sample of people.  A sample is just a smaller number of representative people selected from the whole target group.  For example if you wanted to know the views of all residents (library users and non-users), it would be more efficient to interview a sample of residents, than to interview everyone.

To make sure that the sample is representative of the target group you will need to select the respondents in a random or systematic way.  This means that everyone in the target group has an equal chance of being selected.  This can be done in a number of ways.  For example if you are interviewing users through the door then once you have finished interviewing one person you should approach the next person who passes you.  If you are doing a telephone interview or a mail out, the address should be selected in a random way- you may also need to decide who you will interview in a chosen household.  Will it be the adult (person 16 years or over) whose birthday falls next?  This is a common way of randomly selecting a household member.  You should never allow respondents to select themselves, for example by leaving the questionnaires in a pile for people to pick up if they want to.  Nor should you let interviewers choose who they want to interview.

Choosing samples requires skill and systematic thinking.  For instance surveying users "through the door" requires knowledge of volumes of users day by day, by hour of the day/night etc.  In these circumstances quota sampling may be best.  This means that if visits during high use times comprise 70% of all visits, then 70% of the sample should be drawn from this high-use group.  If 20% of visits occur at a specialist library, separate from the main location, 20% of the sample should be from that venue.  The essential point is to discuss your sample design with an experienced person.

4.    How to Survey

The way you choose to administer the survey will depend on the type of information that you need, your budget and time frame.  Surveys may be carried out in three ways, through a personal interview, a telephone interview or through self-completion.  Combinations of these three methods are common too.  For example you may select users at a venue, gain their co-operation and then have them fill out surveys at home and post them back to you.

4.1          Personal interview

Personal interviews allow quite complex questions to be addressed.  They allow the interviewer and the respondent to interact and exchange detailed information.  Personal interviews are good for sensitive issues, and for lengthy questionnaires.  They are also good for interviewing a geographically centralised group of people.  The generally accepted maximum length for a personal interview by appointment is one hour.  Interviews where the respondent is intercepted at a venue (eg the library) should be kept short, with a maximum length of about ten minutes but preferably five minutes.  With the resources involved, personal interviews are the most expensive method.

4.2          Telephone

Telephone interviews need to involve a more structured exchange of information than personal interviews.  They are good for interviewing a geographically dispersed group of people.  The generally accepted maximum length of a telephone interview is relatively brief (20 minutes).  They are considerably less expensive to administer than personal interviews.  Telephone surveys take less time to administer than either mail or personal surveys.  They are generally not used for sensitive types of questions (for obvious reasons).

One common overseas criticism of the telephone interview is that low-income households may be under-represented in the sample.  This is because some low-income households do not have telephones.  However, in New Zealand most areas have at least 90% household ownership of telephones with some areas as high as 97%.  What you must ensure is that numbers that have no one at home on the first call are phoned again and again until contact is made with that household.  Otherwise your final sample of people will have too many 'stay-at-homes' in it and their use of libraries may well be different from 'harder-to-contact' people.  Similarly telephone interviewing ought to be done during the late afternoon and in the evenings to try and get people when they are at home.  Galling though it may be for respondents to receive a phone call from a telephone interviewer in the middle of the evening meal or their favourite television programme, in the interests of achieving sample purity from telephone interviewing, this is unavoidable.

4.3     Self Completion

You may decide to ask the respondents to complete their own questionnaire.  This method is attractive to many managers because it is the cheapest method of interviewing.  It can provide a good range of views from existing customers.  However, this method is not ideal if you want to interview a random sample of residents as a whole.  This method can also suffer from a relatively low response rate, which can limit the accuracy of the results.  When this method is used inappropriately the results can be easily discredited.

5.    How Many People Should You Survey?

You will need to decide how many people you want to survey.  This is a balance between the cost, the time and the accuracy of the survey.  For example if a large sample is surveyed then the cost will be high, but the results will be more accurate than for a smaller sample.

In all surveys there is the chance that there will be a difference between the results of the sample of people surveyed and the results that would have been gained if everyone in the target population had been surveyed.  The degree of error can be calculated, and is outlined in the chart below.  The chart shows figures at a 95% confidence level.  That is we are confident that 19 times out of 20 the results will fall within the following margins of error.

The chance of error increases as the sample size decreases.  However note that doubling the sample size does not reduce the margin for error by half (but only one quarter). Very small sample sizes should be treated cautiously.  However it is important to note that a smaller well conducted sample survey can be better than a large, badly conducted one.

The chance of error also increases as the results are more balanced.  For example if you survey 50 users of the library and found that 95% of respondents thought the crime fiction collection was of a very good standard, the margin of error on that result would be only 6%. On the other hand if 50% of the respondents thought that the crime fiction collection was of a very good standard the margin of error would be 13.9%. Since we cannot predict the results of a survey, it is best to estimate the margin of error based on a result of 50% (as shown in the above table).

The amount of error that can be tolerated depends on how the survey results will be used.  For example if the survey is going to be carried out regularly to monitor changes in opinion, then it is important that the margin of error is small so that the change in results is an accurate reflection of reality.  On the other hand if the results are going to be used as a general indication of strength of opinion on an issue, and only dramatic results will be used, then a higher margin of error may be tolerated.  In general, however, samples of less than 150 are too small.

You may want to compare the views of one sub-group of your sample to those of another sub-group.  If you depend on random selection you may end up with a very small number of people in the subgroup, and consequently an unacceptable margin of error.  This problem often occurs when analysing figures for ethnic minorities.  For example Maori people make up about 4% to 20% of the population depending on where in New Zealand you live.  If 1000 people were interviewed we would expect to obtain only 40-200 interviews with Maori people.  The margin of error for the sample as a whole would be 3. 1% while the margin for the Maori sample would be much higher.  In order to compensate for this you can treat Maori as a separate sample.  This means you can interview a greater number of this group, and get more accurate results to analyze separately.  Be careful how you do this; make sure that you get help from someone skilled in weighing data.

6.    What is Non-Response?

You will find that not everyone who you want to interview will respond.  This may be because they are not home, too busy, or do not understand the language of the questionnaire, or are simply not interested.  It is important to minimise non-response to make sure that the results of the survey are truly random.  This is because the opinion of those who reply may well differ from the opinions of those who do not reply.  This would mean that the survey results would not be representative.

It is crucial that you do all you can to reduce non-response.  You should always assure the person of the confidentiality of their views.  The questionnaire should also be as brief and easy to understand as possible.  For phone or personal interviews you can call the person back, or arrange a more suitable time to interview them.  You may want to offer a small incentive to people to reply, such as a voucher for a free coffee.  You should also make sure that the language of the questionnaire can be understood by respondents.  For mail out questionnaires you should write a good covering letter and provide a reply paid envelope.  It is important to send out reminder letters to the non-respondents, and include another copy of the questionnaire.

7.    Where to Survey?

If you are interviewing in person you will need to consider where your survey will be carried out.  This is dependent on the survey method that you have chosen.  You may wish to interview the customers as they come through the door, or you may wish to make appointments to see them.  If you want to carry out a user survey for a facility then it makes sense to interview users as they enter or leave the building.  Alternatively if you want to survey non-users then you will need to select them in another location, obviously by another means.

8.    When to Survey?

You will need to decide when you will run the survey.  This will take into consideration the season, day and time that the survey fieldwork will be carried out.  This will have an impact on the type of people you interview.  It makes sense to choose interview times when the most people that you want to interview will be available.  For example if the survey aims to find out what children think of the library, then it could be carried out in the school holidays. (But be careful about interviewing children.  You really need parental consent or ‘in loco parentis' from a school, so interviewing children about the library may best be done through the schools).

9.    How Long Will the Survey Take?

You may want to estimate the duration of the survey so you can meet your deadlines.  The duration of the survey depends on how long it will take to get the number of responses that are required and to analyze the results.  This is determined by the survey method, the response rate and the resources devoted to the survey.  You can calculate the duration of the survey using the following formula:

Estimate how many surveys could be carried out each day, then divide the total sample size by the number of surveys per day to get the duration of the survey fieldwork.  For example if the required survey sample is 300, and it is estimated that 10 interviews will be carried out each day, then the survey fieldwork should take 30 working days to complete.  Then estimate the number of days it will take to analyze and write up the results, and add this to the total.

Note that conducting a survey from beginning to end takes a market research company between six and eight weeks for a small (say 250) face-to-face interview project You should allow plenty of time; it is unlikely that you could complete a survey of this magnitude in-house in under ten weeks.

10.Privacy Act 1993

You need to be aware of the implications of the Privacy Act 1993 for your survey work.  If you want to identify an individual respondent in your research you must ask their permission.  If you want to contact a respondent again in the future you should ask their permission at the first contact.  If you intend to interview children (under 16 years) by telephone, or in the home you will need to gain the permission of their parent/guardian.  Interviews with children at a venue (i.e. the library) in some circumstances can be carried out with the permission of the child her/himself.  However note that the interviews would need to be conducted in a very public place so that interviewers cannot be suspected of any misadventure.  Apart from these provisions the Act does not restrict research as long as it is used for statistical or research purposes and you are not trying to "make a sale".

11.Who Did You Survey?

You will need to include questions about the characteristics of the respondents.  This information is useful for management purposes.  For example the results may show that older people are less likely to use the library, while school aged people are the highest user group (illustration only). Management may want to make changes to the service in response to this information.  Even if you do not think you will need this type of analysis, the information needs to be collected so you can show that the survey sample is representative of the target population.

The following questions should be included at the end of the questionnaire.  The categories outlined below are the minimum; they can be expanded if more detailed information is needed.

12.How to Design the Questionnaire

You will need to decide on the aim of the survey, and the design of the sample before you attempt to design the survey questionnaire.  Designing a questionnaire is not an easy task.  It requires you to express your questions in a way that is easily understood.  The quality of the questionnaire will determine the quality of information that the survey results will provide.  This section provides a guide to things you will need to consider when designing your questionnaire.

·       Traps to Avoid

Questions need to be simple and easy to understand otherwise errors can occur.  If a question is interpreted in different ways by different respondents or the question is too difficult then you will get misleading results.  Try to use everyday conversational language, and avoid technical or industry terms.  It is important to make sure that the questions are free from bias and they flow logically.  The layout of the questionnaire should be clear and easy to follow.

·       Open-Ended Questions

The first type of interview questions that you will want to consider is "open-ended".  These questions arc exploratory, and can provide new insights into an issue.  They are used to get a wide range, and reasonable depth of opinion.

For example:

If the survey is carried out on the telephone or in person then the interviewer can probe the respondent for clarification:

Open-ended questions also allow you to use a general probe for more information:

·       Close-Ended Questions

The second kind of interview questions you will want to consider are "close ended".  They require the answer to fit into pre-determined categories.  Close-ended questions arc useful for numerical analysis.  It is often helpful to undertake some "pre-research" to help frame your questions and your categories.

Close-ended questions can be used to help identify whether it is appropriate to ask another question.

For example:

Close-ended questions can also lead-in to an open-ended question.

For example:

Close-ended questions can indicate the range of opinion on a subjective statement.

For example:

Things to remember about closed questions:

·       Include a "don't know" category where relevant (except in self completion surveys).

·       Include an "other' category, and provide an opportunity for explanation.

·       Make sure that the options that are presented do not overlap (that is, they are mutually exclusive).  For example the following options enable respondents who visited four times to select both the third and fourth categories.

This is a bad question.

13.Is This a Good Survey?

·        How to Review the Survey

You will need to test the survey with users to make sure that it is all in order.  It will save a lot of time and money if you identify any problem with the questionnaire or method before the survey itself is carried out.

The questionnaire should be reviewed three times.  It is useful to get colleagues to do the survey in the first instance to identify any weaknesses.  For each question it is useful to ask:

Will this question provide information that you need?

Are respondents likely to be willing and able to answer this question?

Is there any way that this question can be made easier for respondents to understand?

The second review should be done by people who are like the respondents.  This can involve getting people to answer the questionnaire while a researcher looks on, observing any difficulty that the respondents have with the questions.  An alternative approach is for the researcher to go over the completed questionnaire and discuss any obvious errors with the respondents.

The third review should be a full pilot for the survey to ensure that it runs smoothly.  A pilot survey is a dress rehearsal.  You have the opportunity to test the whole survey process to ensure there are no problems, get an estimate of the time the questionnaire takes, and test instructions.  Typically pilots are conducted with five to ten respondents.

14.Keeping Records

·       Recording the Survey Method

When you report the results of the survey you should outline the survey method you have used in detail so that anyone reading the report can understand it.  This should include details about the design of the sample: that is whom you surveyed, how you surveyed them, and when and where they were surveyed.  If the survey is carried out again in future years the same method should be used, so that the results can be legitimately compared year to year.  It is also important to state the margin of error for the survey results, so that the reader can see whether the results are significant or not (see section 5 How many people should you survey?).

You will also need to include in your report the following details on the response rate:

a.  The target sample size

b.  The total number contacted

c.  The total number of actual respondents

The response rate is the percentage of the target sample size who became respondents.  This can be calculated by dividing the total number of respondents by the target sample size, and multiplying that by 100.

·     Recording the Survey Results

You will need to consider how the survey results will be recorded at the time that you design the survey.  As discussed above questions fall into two categories, open-ended and close-ended questions.  Close-ended questions already have codes which relate to the defined categories.  Open-ended questions will need to be recorded and coded later.  You will need to develop a system for counting the numbers of responses in each category.

15.What Do the Results Mean?

Your decision about which results to analyze and how to analyze them will be determined by the objectives of your survey.  There are many ways to interpret survey results.  In most cases you will find that a relatively basic analysis can provide you with the information that you will need.  A simple way of analysing survey results is outlined below.

·     How to Analyze the Results

The most common way to analyze survey results is to count the number of responses for each variable, and express this number as a percentage of the total sample. The table below shows fictitious ratings of satisfaction with a local library

The crucial results here are that the library is perceived as performing rather poorly or at best. adequately by the majority of patrons.  Just under half (41%) have chosen "adequate" while just over one quarter (28%) have selected "poor".  "Good" or better is the view of only one quarter (4% +22%).

Now it would be beneficial to explore these results further.  Who exactly is disenchanted with the Library - is it everyone equally or (for instance) are there differences by gender, by age, by frequency of library use etc?

In the analysis shown here, 36% of women thought the library was performing poorly or very poorly compared to 26% of men.  Using the error margin details discussed above (section 5), you can see that a 10 percent point difference between these two groups, with sample sizes of 443(men) and 558 (women), will yield a statistically significant result.  That is, the 10 percentage point difference is not due to chance but due to some real difference in perception of performance.  Conversely, 31% of men and 2 1% of women thought that the local library's performance was "good" or "very good" - as the above discussion shows, this percentage difference is real rather than just the result of a small sample or chance.

16.Using the Results

·       Management Decision Making

The main point of carrying out a survey is to gain objective information from users (and sometimes non-users) about the library's priorities, services and facilities.  It is crucial that the survey results are taken seriously by management, and used to guide future decisions.  When analysis of a survey is carried out it is important to sit down and consider any recommendations for action that arise from the results.  For example if you find out that residents are not aware of your service then you should propose options to management for improvements.  These could include a proposed marketing campaign targeted at those groups of residents with low awareness.

In addition to this, relevant Council Committee should also receive a report on the results of the survey.  This will keep Councillors up to date with the latest information about the library.

·       Communicating the Results

It is important to make sure that the results of the survey are made available to anyone who is interested.  You may particularly want to inform respondents about the results, as one way of acknowledging their input.  For example you could put up the results on the notice board of relevant libraries.  If the results are particularly interesting you may want to work with management to write a press release for local newspapers.