ETSI TS 102 852 V1.1.2 (2011-07)

User Group; Quality of ICT Services; Assessment process of the QoS parameters of the customer relationship stages

RTS/USER-00036

General Information

• Status: Published
• Publication Date: 10-Jul-2011
• Technical Committee: USER - User Group
• Current Stage: 12 - Completion
• Due Date: 13-Jul-2011
• Completion Date: 11-Jul-2011

Standards Content (Sample)

Technical Specification
User Group;
Quality of ICT Services;
Assessment process of the QoS parameters of
the customer relationship stages

2 ETSI TS 102 852 V1.1.2 (2011-07)

Reference
RTS/USER-0036
Keywords
CTS, performance, QoS, user
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2011.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI TS 102 852 V1.1.2 (2011-07)
Contents
Intellectual Property Rights . 5
Foreword . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions and abbreviations . 6
3.1 Definitions . 6
3.2 Abbreviations . 7
4 Definition of a QoS assessment campaign, aims and purposes . 7
4.1 General . 7
4.2 Questions to be answered when designing a QoS assessment campaign . 7
4.3 Tasks for the campaign . 8
5 Boundary conditions for campaign . 8
5.1 Customer survey . 9
5.2 Expert panel . 10
5.3 Service Provider Data Analysis . 11
5.4 Samples sizes and examples (excerpt from EG 202 843) . 11
6 Preparation of the campaign . 12
6.1 Customer Survey . 13
6.2 Expert panel . 14
6.3 Analysis of SP data. 14
7 Execution of the campaign . 14
7.1 Customer survey . 14
7.2 Expert panel . 14
7.3 Analysis of SP data. 15
8 Data filtering . 15
9 Aggregation of results . 15
10 Analysis of the results of the assessment campaign . 15
11 Report . 16
Annex A (normative): Parameter overview . 17
Annex B (normative): Questionnaires for customer survey and expert panel . 23
B.1 Opinion Rating (OR) . 23
B.2 Rate (%) . 24
B.3 Time (t) . 24
B.4 Number (N) . 25
B.5 Other elements to be added in the questionnaires . 25
B.5.1 Pre-conditions . 25
B.5.2 Open questions . 25
B.6 Validation of the questionnaire . 26
B.7 Questionnaires addressed to attendees . 26
ETSI
4 ETSI TS 102 852 V1.1.2 (2011-07)
B.8 Examples from the experiment made by STF 374 . 26
Annex C (informative): Questions related to the user's background . 27
C.1 Questions related to the user's background . 27
C.1.1 Personal data . 27
C.1.2 How often do you use ICT services on an average (several possible answers)? . 27
C.2 What kind of ICT are you contracted with your Service Providers - (several possible answers)? . 28
C.3 What are the terminals you are currently using (several possible answers)? . 28
C.4 What kind of access (several possible answers) for internet, if you know them? . 28
C.5 Service providers . 29
C.5.1 With how many service providers do you have contracts for your services? . 29
C.5.2 Have you moved at least once from a service Provider to another one? . 29
C.5.3 Do you use ICT for . 29
C.6 Do you have any additional information to provide? . 29
Annex D (informative): Example of a campaign performed by STF 374 . 30
Annex E (informative): Examples of tasks for Expert panels . 34
Annex F (informative): Bibliography . 46
History . 47

ETSI
5 ETSI TS 102 852 V1.1.2 (2011-07)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Technical Specification (TS) has been produced by ETSI User Group (USER).
Introduction
The present document provides the requirements needed to ensure that QoS information is assessed according to the
best practices as detailed in EG 202 843 [i.1]. It is closely linked to:
• EG 202 843 [i.1] that defines parameters and basic information which are universally applicable to QoS
assessment. One aim here is to keep the parameter definitions stable and complete for any kind of application.
• TS 102 844 [1] that defines when, how and by whom a conformity assessment audit of the QoS assessment
shall be conducted, if such conformity assessment is desired.
ETSI
6 ETSI TS 102 852 V1.1.2 (2011-07)
1 Scope
The present document defines a common set of test sequences that are applicable by any stakeholder in order to provide
results that are objective, reproducible and comparable.
It details - in a generic way - the assessment sequences for the 4 types of parameters: Time, Percentage, Number and
Opinion Ratings as given in EG 202 843 [i.1].
The present document also provides examples of a set of dedicated assessment sequences for parameters of selected
stages of the customer relationship course (QoS Assessment process).
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
reference document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
[1] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment;
Requirements for bodies providing QoS assessments and surveys".
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI EG 202 843: "User Group; Quality of ICT Services; Definitions and Methods for Assessing
the QoS parameters of the Customer Relationship Stages other than utilization".
[i.2] ITU-T Recommendation P.832: "Subjective performance evaluation of hands-free terminals".
[i.3] ETSI EG 202 057: "Speech and multimedia Transmission Quality (STQ); User related QoS
parameter definitions and measurements".
[i.4] ETSI EG 202 009: "User Group; Quality of telecom services".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
test method: assessment method of QoS parameters carried out by a Quality of Service Assessment Party (QoSAP)
providing QoS audits and surveys based the assessment of parameters described in EG 202 843 [i.1]
ETSI
7 ETSI TS 102 852 V1.1.2 (2011-07)
test sequence: detailed procedure of the test method applying to the types of parameters as described in
EG 202 843 [i.1]
NOTE: Test in the context of the present document and EG 202 843 [i.1] has a different meaning from
compliance or interoperability testing.
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
ADSL Asymetric Digital Subscriber Line
CS Customer Survey
DSL Digital Subscriber Line
EP Expert Panel
FTTH Fiber To The Home
IPTV Internet Protocol Television
ISDN Integrated Services Digital Network
KPI Key Performance Indicator
OR Opinion Rating
PDA Personal Digital Assistant
PI Preliminary Information
PSTN Public Switched Telephone Network
QoS Quality of Service
QoSAP Quality of Service Assessment Party
SMS Short Message Service
SP Service Provider
SPDA Service Provider Data Analysis
SPO Service Provider Oriented (parameter)
TMF Telemanagement Forum
VoD Video on Demand
VoIP Voice over IP
4 Definition of a QoS assessment campaign, aims and
purposes
Prior to executing a campaign, the concrete aim of the campaign shall be defined and documented. The campaign may
be initiated by any stakeholder (SP, Consumer association, regulatory bodies, etc.). The design of the campaign could
depend on the specific objectives of a stakeholder. The next clauses define generic principles to design a campaign. The
QoS parameters shall be implemented according to EG 202 843 [i.1].
4.1 General
The EG 202 843 [i.1] defines the different stages of the customer relationship, the parameters and the preferred methods
to assess them. Table A.1 summarizes the main information and recommends the preferred approaches detailed in
EG 202 843 [i.1]: Customer Survey, Expert Panel and SP Data Analysis.
Defining the aim of a campaign means that several choices and decisions have to be carried out. One of the important
initial tasks is to define the perimeter of the campaign and to define the profile of the panels to participate to the
campaign.
4.2 Questions to be answered when designing a QoS
assessment campaign
When designing a QoS assessment campaign, a set of questions has to be answered, which determine the elements to be
taken into account within the assessment campaign.
ETSI
8 ETSI TS 102 852 V1.1.2 (2011-07)
Several Service Providers may be involved in the assessment campaign. For each of them the following questions shall
be answered:
• What are the selected stages of the customer relationship lifecycle, as defined in EG 202 843 [i.1]?
• What is the list of services within the SP offers that will be assessed?
• Phases can be used as modules; different modules building the complete interrogation.
• How many Service Providers will be involved in the assessment campaign?
• Is the assessment campaign defined as a Benchmarking? -Direct comparison of different providers
(e.g. comparison of success rates or comparison of time parameters across different providers).
• Will the assessment campaign be a "one shot" or be made periodically?
• What type of approach has to be implemented: customer survey, expert panel, SP Data, Analysis or a mix of
them?
• What customer group has to be involved (For customer survey and expert panel, the profiles and the number of
participants have to be specified).
4.3 Tasks for the campaign
A table defining a set of tasks to be done by the Expert Panel is provided in informative annex E.
• For the expert panel the tasks are designed specifically in order to ensure all the providers and/or services are
treated in a similar way and to reduce as much as possible the time spent in collaborative works between the
experts.
The customers involved in the customer survey shall be currently using the services and they have to answer a set of
questions.
• For customer survey real example of a customer survey campaign including several customer relationship
stages is provided in annex D and could be used as an informative reference when defining a campaign.
• For the approach based on SP Data Analysis the design of the test should take into account the type of data
made available by each service provider.
5 Boundary conditions for campaign
For recurrent and benchmarking campaigns the conditions have to be clearly defined in order to make results
comparable and reproducible. This includes in particular the selection of expert panels or customers participating in the
surveys, the test duration, the media to access participants, etc.
When preparing an assessment campaign related to the customer relationship course all the variables that may influence
the assessment outcomes shall be defined. One main reason for detailing all the parameters is to give the possibility for
other parties to reproduce comparable assessments. As soon as one of the variables is changed, the campaign might
characterise a different assessment with deviations in the results.
All the parameters that define an assessment campaign have to be checked carefully and listed in the assessment plan, in
particular:
• The parameters that are retained within each stage have to be defined for each stage of the customer
relationship life cycle. This could include dedicated part of customer relationship cycle, e.g. only DSL clients,
only roaming issues of business users, etc.
• As EG 202 843 [i.1] defines three types of approaches, "Expert panel", "Customer panel" or "Service Provider
data Analysis", the most appropriate type must be chosen for the different parameters (see annex A). For each
parameter included in the campaign, the option chosen has to be defined and justified.
ETSI
9 ETSI TS 102 852 V1.1.2 (2011-07)
• If the campaign includes Customer surveys, the appropriate customer groups to be involved in the assessment
shall be determined. In particular it shall be defined if there is a focus on private and/or business customers.
The number of customers involved in the campaign shall also be established to ensure statistically valid
results. It shall be checked that the customers involved in the assessment sessions will not participate in more
than 2 sessions per year.
• If the campaign includes expert panels, the characteristics/profiles of the experts to be involved shall be
defined.
• The assessment plan shall also define the media that will be used to address the attendees and the media that
are foreseen for the feedback of the attendees.
• The assessment plan shall define the campaign duration, and if the campaign is carried out on a periodical
basis and the periodicity of the sessions.
• If questionnaires are addressed to customers, it shall be defined if and when reminder messages could be sent
during the runtime of the assessment.
• It shall be checked if there is a special geographical area of interest, e.g. handling of DSL contracts in rural
areas where no broadband data service is available due to physical constraints.
• For customer surveys, it shall be defined what kind of methods will be used: e.g. telephone interview, mailing
activity, online questionnaire, direct interview in a shopping mall.
• For expert panel it shall be defined if the experts will be questioned in-situ or at distance, and if they will do it
individually or all together (in such a case, preferably a physical meeting).
• The operational team (QoSAP) that will design and run the assessment shall be defined and documented. This
shall be defined for customer surveys, expert panels and analysis of SP Data.
• It shall be checked if the operational team (QoSAP) needs to involve other companies or departments, e.g. call
centres for phone interrogations, companies for recruiting panels, companies for campaign execution.
For Expert panel and Customer survey even if the approaches may be different, the questionnaires/measurement
methods may be rather similar.
In particular, questionnaires shall be designed to select customers and experts to be involved in the assessment sessions.
These questionnaires should be based on the contents of annex C.
It shall be taken into account that the customers may only reply for services or providers they have actually
experienced.
For experts (or experienced/trained users) it is needed to design the tasks so that all the service providers are assessed in
similar/equal conditions and equal footing. The experts are participating in the assessment session for a limited period
of time and they shall complete the tasks which are defined within the assessment sequences.
5.1 Customer survey
To get a broader basis of feedback, customer surveys (panels) can be used. A customer panel consists of common
customers of products or services. The customers should be familiar with the topic they are surveyed for, but without
reaching an expert level. The selection should be based on replies to the relevant questions defined in annex C. The
selection may also be done by Service providers, customers groups, etc.
In many cases, specialized institutes are engaged to deal with the panel recruitment. This is based on the fact that either
a well-defined part of the population should be taken into consideration (e.g. only females aged 25 to 35 years with a
certain net household income) or that the selected group of customers should be representative for the complete
population of this country or for the complete population of customers of a service provider.
When selecting the customer panel it may be needed to ask questions related to the user's background, such as defined
in annex C.
To ensure relevant statistical results, the number of customers shall be important. The level of customer attendance to
ensure the desired level of statistical relevance of data is defined in EG 202 843 [i.1] and copied in clause 5.4 of the
present document.
ETSI
10 ETSI TS 102 852 V1.1.2 (2011-07)
When designing the assessment event it is needed to take into account that additional expenses are generated by the
involvement of market research institutions.
Customer survey interrogations need currently a longer period of time than for the expert panel (up to several weeks).
When Opinion Rating (OR) is sought via both a customer survey and an expert panel, there may be discrepancy
between the findings of these differing means. Where the difference is significant, reason for this discrepancy should be
investigated and any necessary changes incorporated either to the panel's ratings or the way the customer survey is
carried out.
When questionnaires are sent to an email list of customers (or by postal mail) or when customers are asked by phone
calls, the percentage of replies may be very low compared to the whole customer list. This has to be taken into account
to ensure a high level of statistical relevance of the results.
5.2 Expert panel
An expert panel is defined as a group of experts, who are very familiar (or professional) with the topic of interest. The
expert panel evaluates the topic of interest and gives expert opinion on it.
For some experiments, experts may be replaced by experienced or trained customers/users. To distinguish experts and
experienced/trained users, the following definitions have been adapted from ITU-T Recommendation P.832 [i.2]:
• Experienced/trained users: Experienced/trained users (for the purpose of customer relationship QoS
evaluation) are experienced in QoS testing and analysing the quality of some stages of the customer
relationship, are not individuals who routinely conduct such assessments. Experienced/trained users are able to
describe their subjective impressions on the different stages in detail. However, experienced/trained users
neither have a background in technical implementations of customer relationship stages covered by
EG 202 843 [i.1] nor do they have detailed knowledge of the influence of the parameters on the quality
perceived by the customers.
• Experts: Experts (for the purpose of customer relationship QoS evaluation) are experienced in QoS testing
and analysing the quality of the different stages of the customer relationship. Experts are able to describe all
the events, tasks and deliverables of the customer relationship in detail and are able to implement the
assessment methods or to analyse the results as described in EG 202 843 [i.1] and to analyse the specific
impairments in the processes. They are able to describe their impressions in detail. They have a background in
technical implementations of the customer relationship stages and/or do have detailed knowledge of the
influence of customer relationship implementations on QoS perceived by the customers. Experts may be used
for all types of assessments defined in EG 202 843 [i.1]. Typically the expert's judgement is validated by
untrained users representing the average user group the set is intended to be used for.
In case where the number of experts cannot be reached, experienced/trained users may complete the team or the group.
However, if experts and experienced/trained users are mixed within an experiment care should be taken to avoid experts
masking the feedbacks from experienced/trained users, in particular when all the participants meet together.
Ideally, the selected experts bring a broad theoretical background and practical experience as well as a longer period of
personal knowledge with them. Besides that the selection of experts should take into consideration that all relevant
aspects of the examined topic are covered by the combination of experts within the panel.
The number of experts required to address a certain topic should be limited (currently between 10 and 20).
A questionnaire to select expert panel is defined in annex C:
• Selection process and criteria for selection are very important because the quality of the experts may influence
severely the quality of the results.
• It is also important that all required experts may be together at the same place and time (or over a short period
of time), or by high quality conference systems.
NOTE: It is needed to plan for expenses generated by the involvement of experts.
ETSI
11 ETSI TS 102 852 V1.1.2 (2011-07)
For the expert panel, two different approaches can be undertaken, depending of the relationship stage:
• In a first step each expert realises specific tasks individually and provides replies to questionnaires and
comments based on their own knowledge and perceptions; in a second step they meet together to exchange on
the results or observations obtained.
• The task is done collectively on the basis of materials made available to all the experts during a specific
meeting. In such a case feedback to one specific subject can be collected rather quickly (during an experts'
meeting).
As Experts are used as highly-trained customers they may reflect customers' point of view and may also comment
together on the results.
Studies carried out on particular QoS aspects like assessment of call centre QoS made using "mystery calls" or QoS of
mobile communications by human operators belongs to this category of data source. The experts involved in the
"mystery calls" should act as normal customers but shall ask pertinent questions to obtain detailed answers.
5.3 Service Provider Data Analysis
For certain customer relevant processes, service providers (SPs) may have available customer records for their own
purposes or due to regulatory requirements. In these cases such data may be used for the determination of customer
relevant parameters as well but in a well controlled process.
NOTE: It appears that some parameters defined by Telemanagement Forum (TMF) may be used for such
approaches. Some further investigations will be needed to achieve this.
Before using SP Data for an analysis, it shall be checked that the parameters and measurement methods defined in
EG 202 843 [i.1] apply for the data collected. If the measurement conditions implemented by the Service Providers
differ from those available in EG 202 843 [i.1], it should be checked that the data may be used without any trouble. The
service provider shall document the measurement conditions for the data collection and shall ensure that the purpose
and conditions are compliant with the principles defined EG 202 843 [i.1]. Prior to implementing such a process it will
be checked that the SP Data are well correlated with parameters defined in EG 202 843 [i.1].
It could be expected that such an approach should reduce the data generation since the data is available from the usual
day-to-day business. It could also be associated to customer surveys or expert panels.
A large amount of data sets may be available (mass data), depending on the number of customers contracted by the SP
and depending on their activity.
For such an approach the service providers will be an active actor of the process.
5.4 Samples sizes and examples (excerpt from EG 202 843)
Besides the different nature of the mentioned data sources, the number of available samples for each of these data
sources may also differ heavily, as indicated in EG 202 843 [i.1].
• To assess a special topic, only a few but highly trained experts are required. This leads to a high quality
feedback, but includes also very limited number of information.
EXAMPLE 1: 15 experts are requested to assess the "Integrity of Complaint Resolution". The outcome will be 15
different opinions on corresponding scale.
• The assessment of topics which are more common to all customers and which do not require special expertise
allows the involvement of a higher number of customers.
EXAMPLE 2: Replies from 150 customers of SP A who complained about a certain matter are selected to give
their feedback on the "Customer Perception of the Complaint Management".
Here, the quality of the feedback will not be on expert level, but represents the customer perception very
clearly. Furthermore, the number of samples is higher than in the first case which improves the data basis for
statistical operations.
ETSI
12 ETSI TS 102 852 V1.1.2 (2011-07)
• Finally, if mass data from service provider's internal processes can be assessed, there are two advantages: The
weight of each data set on the overall result is negligible, and most of the data will be measured objectively.
EXAMPLE 3: SP B delivers 10 000 data sets which allow to determine the parameter "Time for alteration" on a
very broad basis.
6 Preparation of the campaign
After defining the basis of the assessment by fixing the basic boundary conditions, the operational phase should be
prepared, depending on the assessment approaches chosen.
The following should be born in mind when planning surveys.
Basically there are three methods of carrying out a customer survey:
1) questionnaire posted to the interviewees;
2) questionnaire administered by an interviewer over telephone; and
3) administering questionnaire on a face to face basis between the interviewer and the interviewee.
Each method has its merits and disadvantages. These are outlined briefly here.
Questionnaire posted to the interviewee
Respondents are asked to fill in the quality ratings for the performance parameters.
Advantages:
Questionnaire is a powerful tool to capture customers' performance ratings. This method of administering a survey is
also the cheapest as very little manpower is involved.
Disadvantages:
In filling in questionnaires, the respondent does not usually have the means to seek clarification on questions they do
not understand. This could result in vague answers. Some parts of the questionnaires may be left blank. There may also
be some questions the respondents may have misunderstood.
The response rate is low. In general these vary from below 5 % to upper 30's % for user club members. This means a
very large sample size has to be sent the questionnaire to obtain a statistically valid number of responses. Additionally
the answers should be carefully analysed to ensure that the replies provided clearly indicate that the customer has
understood the question. It must be borne in mind that some customers fill in the form without much thought. Indeed in
certain psychological questionnaires it is common to insert a "lie detector" question to ensure that the answers provided
are consistent to the person's beliefs in providing answers.
Questionnaire administered by an interviewer over telephone
Advantages:
Telephone questionnaires are a compromise between postal questionnaire and face to face interview. The interviewer
can ensure to a greater extent than the postal survey that the customer has understood the question. There is scope for
the customer to clarify points she/he is unsure of. The response rate is also much higher than those of postal survey.
Disadvantages:
The principal shortcoming of this type of interview is the absence of body language of the respondent. This limits the
understanding of the questioner of the respondent's concerns.
Cost of administering telephone interviews are higher due to the manpower costs comprising overhead and training
costs.
ETSI
13 ETSI TS 102 852 V1.1.2 (2011-07)
Administering questionnaire on a face to face basis between the interviewer and the interviewee
Advantages:
Face to face interviews, with a questionnaire, should provide the ideal alternative to the postal and telephone
questionnaire. Some of the shortcomings mentioned in the previous methods can be eliminated.
Face to face interviews provide the most credible answers and therefore the confidence in their accuracy and relevance
is very high. This type of survey is usually kept aside for large organisations.
Disadvantages:
Face to face interviews are time consuming and make a high demand on resources:
NOTE: The sample size required for postal questionnaire is the highest, with telephone interviews requiring fewer
samples and the method of personal interviews requiring least samples for a given level of confidence.
6.1 Customer Survey
When the area of the campaign has been defined and the questionnaire built up it is important to prepare all the
materials that will be addressed to the potential participants (by post, e mail, phone call, etc.).
As indicated in annex B, the questionnaire will be addressed to the selected participants. Care should be taken when
defining the messages associated or preceding the questionnaire. These messages shall provide confidence and show the
professionalism of the process.
It should also take care of security matters, in particular when using online questionnaires.
A pre-test shall be organised to ensure or improve the overall quality of the activity:
• Prior to official assessment campaign the pre-test phase shall be organised in the exact same way as the real
test but with reduced number of participants. All the comments and issues received during this pretest phase
shall be considered to finalise the assessment campaign.
To save cost and time it is preferable to prepare automated evaluation routines that can be applied during the evaluation
phase and after the evaluation phase has finished.
During the experiment it is strongly recommended to monitor the development of the results; this may prevent issues
from only being discovered after finishing the assessment campaign, e.g. if certain data is entered by the customers but
does not reach the database due to minor errors in scripting.
The QoSAP shall inform the participants on the availability of an hotline that may be reached by phone or mail and to
indicate the delay expected for replies to messages addressed to the hot line.
Care shall be taken to ensure the availability of all required resources, e.g. agents in the call center, mailing and
computer systems for sending out invitation emails and for collecting feedback information.
The QoSAP shall prepare a monitoring of all relevant activities within the evaluation process, at least:
• To set up a quick reaction team to allow last minute changes within the assessment if this scenario might be
useful.
• To monitor all kind of user activity on the system (mainly for online questionnaires):
- To prevent running into overload situations (processor power, storage capacity of database, data links,
etc.).
- To prevent participants from misusing the system. In particular the injection of query statements to
databases and the addition of remotely controlled frames to web applications should be checked in
advance.
The QoSAP shall define a backup strategy to prevent from data loss.
ETSI
14 ETSI TS 102 852 V1.1.2 (2011-07)
6.2 Expert panel
For Expert panel and Customer survey the approaches are different. The expert panel works on a task-oriented process,
as defined in annex E. However, some questions addressed to experts may be similar or based on examples available in
annex D.
Due to the reduced number of experts compared to customers involved in a survey, the process may be simplified.
However processes similar to customer surveys may be used for communications with the experts and to collect the
data.
When experts are invited to meet physically or through high quality teleconference systems, the scenario of the meeting
shall be defined very precisely, in particular the tasks to be done collectively and it shall be needed to ensure that all the
experts are placed in similar conditions and on equal footing (all the experts have to express themselves; it shall be
avoided that one or some experts take the lead of the discussions and impose their views). For additional analysis, the
meeting may be recorded (audio and/or video).
6.3 Analysis of SP data
After having checked that the SP data may be used to assess some parameters defined in EG 202 843 [i.1], the service
provider shall provide data in formats that are defined by the QoSAP.
7 Execution of the campaign
The execution of the campaign includes:
• The execution of the monitoring (see annex B).
• The activation of the hotline team for competent and quick resolution of issues.
7.1 Customer survey
It is not intended to invite customers to any meeting to discuss the comments or the additional information.
After the questionnaire has been sent to the participants, they have to reply before the defined deadline (usually two
weeks). It may be relevant to send a reminder after one week to the potential participants who did not already answer.
When sending the questionnaires by email, several preventive actions have to be taken to avoid unsecure exchanges. In
particular when using online questionnaire, the access to this questionnaire should be given in a two step process
including email exchanges. An example is given in annex D.
It is also important during the campaign execution to ensure a daily hotline to reply to questions or remarks from the
participants. The media used for the hotline should be as close as possible to the media used for the questionnaire.
7.2 Expert panel
The experts will be invited to a meeting to discuss the results they individually obtained or to work all together in a
collaborative way.
If experts have to work individually before the meeting, they shall have to complete these actions prior to the meeting.
The duration of these preliminary actions will be defined according to the tasks themselves. Prior to these actions the
QoSAP shall have prepared all the materials needed by the experts. During these periods the QoSAP shall be able to
answer quickly to questions or messages from the experts and to do all the relevant actions to solve potential issues.
The date and duration of the meeting will be known sufficiently in advance and all the experts shall participate.
The chair of the meeting shall be experienced in this kind of meeting.
ETSI
15 ETSI TS 102 852 V1.1.2 (2011-07)
7.3 Analysis of SP data
The tables provided by the QoSAP have to be filled by the service provider.
8 Data filtering
After the operation phase has been concluded, all data should be validated before doing computational assessments.
This comprises different tasks like:
• Removing all unrealistic data, e.g. fake users who want to disturb your assessment.
• Removing data which was entered due to system problems, e.g. values out of range.
Before doing any manipulations on the data basis, all available data should be stored in a safe backup. If one of the
following steps destroys data or manipulates data in an unintended manner, there is still a chance to recover from these
situations.
If data has to be removed from the data basis, this should be done by setting an invalid marker to the data instead of
physically deleting the data. The ratio of invalidated and validated data sets can be interpreted as an additional quality
measure for the evaluation.
9 Aggregation of results
According to the pre-specified rules, data now can be aggregated. The aim of this activity is to transform the retrieved
data to key performance indi
...