ETSI TR 103 114 V1.1.1 (2013-10)

Speech and multimedia Transmission Quality (STQ); QoS Parameters and measurement methodology for Smartphones

DTR/STQ-00194m

General Information

• Status: Published
• Publication Date: 17-Oct-2013
• Technical Committee: STQ MOBILE - STQ Mobile
• Current Stage: 12 - Completion
• Due Date: 12-Nov-2013
• Completion Date: 18-Oct-2013

Standards Content (Sample)

Technical Report
Speech and multimedia Transmission Quality (STQ);
QoS Parameters and measurement methodology for
Smartphones
2 ETSI TR 103 114 V1.1.1 (2013-10)

Reference
DTR/STQ-00194m
Keywords
3G, GSM, multi service testing, QoS, smartphone
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2013.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI TR 103 114 V1.1.1 (2013-10)
Contents
Intellectual Property Rights . 4
Foreword . 4
Introduction . 4
1 Scope . 5
2 References . 5
2.1 Normative references . 5
2.2 Informative references . 5
3 Definitions and abbreviations . 5
3.1 Definitions . 5
3.2 Abbreviations . 5
4 Smartphones as a QoS Test Environment . 6
4.1 Boundary conditions of smartphones . 6
4.2 Popular Operating Systems used on Smartphones . 7 ®
4.2.1 Android . 7
TM
4.2.2 iOS . 7
SM
4.2.3 Windows Phone 8 . 7
TM
4.2.4 Blackberry OS . 8 ®
4.2.5 Symbian . 8
5 Limitations of Smartphones . 8
5.1 Device Limitations . 8
5.2 Operating System Limitations . 8
5.3 Approximations due to working on application layer . 9
5.4 Attended, unattended and automated Operations . 9
5.5 Further Performance Considerations . 9
6 Basic Settings for QoS Assessments . 10
6.1 General . 10
6.2 Trigger points . 10
6.3 Timeouts . 10
7 How to determine QoS parameters in smartphone QoS testing . 11
7.1 Web Browsing HTTP . 11
7.1.1 HTTP Service Non-Accessibility [%]. 11
7.1.1.1 Trigger Points . 11
7.1.1.2 OS specific mappings . 11 ®
7.1.1.3 Android OS . 11
8 Test Methodology . 11
9 Multi-Service Testing . 12
9.1 Basic scenarios for stand-alone devices . 12
9.1.1 Basic principle . 12
9.1.2 Characteristics and limitations . 14
9.1.3 Example: Data as a background service ("CS in PS") . 15
9.1.4 Example: Voice as a background service ("PS in CS") . 15
9.1.5 Multi-service QoS parameter constellations . 16
9.2 Extended scenarios for device pairs . 16
9.3 Download and Upload Throughput Tests in parallel . 17
History . 18

ETSI
4 ETSI TR 103 114 V1.1.1 (2013-10)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
Introduction
Smartphones can be used as a platform for the execution of Quality of Service measurements. However, compared to
host based systems, some restrictions apply.
These restrictions of smartphones are not only dealing with limited processing power and memory, but also with the
availability of information to execute QoS measurements. The access to the operating system, the provided functionality
to gain network and service related information might make a difference when trying to implement QoS testing within
smartphone environments.
In detail, differences and necessary changes compared to host based systems are pointed out. However, the well-defined
QoS environment already existing for host based systems should be reused wherever possible. Smartphone specific
procedures and settings should complement or override existing definitions if necessary.
A specialty of smartphones is the parallel usage of different services. The so called multi-service testing describes a
complex scenario where at least two services are used in parallel. This situation has to be considered when discussing
QoS measures and results.
ETSI
5 ETSI TR 103 114 V1.1.1 (2013-10)
1 Scope
The present document is intended to discuss the specialties of Quality of Service testing procedures executed on
smartphones and to give according guidance of how to use smartphones for successful QoS testing.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
3 Definitions and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
App: application that can be added by the user to his smartphone
NOTE: Applications have at least one component which is running on user level and which is visible to the user.
host based system: computing system similar to a personal computer acting as controlling entity for the
communications device itself (e.g. a USB data stick or another kind of mobile device)
smartphone: mobile device based on an operating system which can be programmatically controlled via a
programming interface, in combination with the possibility to run applications on user level
NOTE: The form factor is smaller than that for tablets.
tablet: mobile device which has a display with touch functionality and a display size of about 20 cm or more
ETSI
6 ETSI TR 103 114 V1.1.1 (2013-10)
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
App Application
ARM Advanced RISC Machines
CS Circuit Switched
DL Downlink
FACH Forward Access Channel
GUI Graphical User Interface
HTTP HyperText Transport Protocol
IP Internet Protocol
LAC Location Area Code
MOS Mean Opinion Score
OS Operating System
POSIX Portable Operating System Interface
PS Packet Switched
QoS Quality of Service
RAB Radio Access Bearer
TM TM
RIM Research In Motion, earlier name of the company BlackBerry
SD Secure Digital
SMS Short Message Service
TCP Transport Control Protocol
UE User Equipment
UL Uplink
URL Uniform Resource Locator
4 Smartphones as a QoS Test Environment
This clause emphasizes the special characteristics of smartphones with respect to QoS tests and presents some of the
most popular operating systems.
4.1 Boundary conditions of smartphones
This clause describes the different boundary conditions of smartphones to acquire test related information. Different
types of smartphones might be used for that purpose:
• Smartphones sold by manufacturers or providers allow to give an impression how normal users experience the
quality of the offered services. Since no adaptations besides the addition of the application used for QoS tests
are made, the general behaviour of such a device is expected to be nearly unchanged.
• Several devices allow to replace the delivered operating system by customized versions. Especially in the area
of Linux based operating systems, so called custom ROMs (Read Only Memory) are popular to extend the
user's rights on his own device. By applying these customized operating systems, additional tools like tcpdump
can be executed to gather more detailed information, e.g. related to the network traffic of the device. However,
the changed operating system as well as additional tools can have an influence on the performance of the
services that are under consideration.
• To achieve a deeper knowledge about internal data processing, additional and mainly manufacturer dependent
libraries can be added, e.g. to trace layer 3 signalling data. By adding these components, the mobile devices
are turning more and more into specialized measurement engines. Especially in relation to the remaining
performance for services under test this has to be considered and to be checked in depth. Performance checks
and comparisons before and after applying additional tool libraries are strongly recommended to assure that
the influence on the Quality of Service experienced by the user is minimized.
ETSI
7 ETSI TR 103 114 V1.1.1 (2013-10)
For Quality of Service testing, many smartphones bring lots of capabilities with them which support the testing of QoS
in nomadic or mobile scenarios:
• Location awareness: Smartphones often can detect their own location by either using satellite navigation
systems or network based location services. Furthermore, integrated position and movement sensors help in
detecting changes of the current location.
• Service support: Smartphones in general support many different services which can be used for QoS testing as
well (e.g. messaging, voice calls, data transmissions, streaming, application specific protocols).
• Mass storage: Most smartphones carry storage capabilities which allow to work autonomously and to upload
collected datasets in a later stage, e.g. if a data connection is available again for this purpose.
• Battery power: The available battery power is one important resource constraint once the smartphone is
disconnected from its charging entity. The trade-off between performance and battery power shows a strong
influence the higher the requested performance of the smartphone is. Therefore, the test design has to be
implemented in an intelligent way to gain the highest possible benefit out of the available energy resources.
4.2 Popular Operating Systems used on Smartphones
Since different operating systems on smartphones reached a remarkable market penetration, this clause gives an
overview over some of their main characteristics related to QoS testing. ®
4.2.1 Android
® TM
The Android operating system is introduced by Google Inc . It evolved to version 4 which is able to handle ®
applications on mobile handsets and tablet devices in parallel. Android is used by many different manufacturers for
their devices. ®
Android implements a Java based virtual machine on top of a Linux kernel. It covers all aspects like power
management, memory management and the life-cycles of the applications. The user access to the Linux environment is
restricted for security reasons so not all operations on the underlying Linux environment can be performed without
modifications.
Alternative vendors of operating systems offer so called Customized ROMs to enhance the pre-defined functionality of ®
the standard firmware and to grant administrator access to the devices. In Android , devices gaining administrator
rights are called to be "rooted". Applications consist of separate activities that may interact with each other. Therefore,
applications are compiled of a set of loosely coupled activities.
Different markets allow the user to download applications to his devices. For developers, all applications could be
installed locally on available devices.
TM
4.2.2 iOS
TM TM
Released by Apple Inc. , iOS (or called iPhone OS in earlier versions) establishes an operating system on top of a
TM
was the first operating system which combined touchscreen functionality with the "always on"
POSIX basis. iOS
TM TM
Internet of mobile devices. iOS is only used on devices release by Apple Inc .
TM
Again, the user's rights are limited by default. Evolving over versions, iOS supports multi-tasking and service
instances in the latest versions. Currently, no information related to WiFi networks is available on application level,
making use of unrooted devices.
® TM
In contrast to Android , all applications have either to be downloaded via the infrastructure of Apple by default or
could be distributed via privately owned web servers in a dedicated company environment. Local installations are
possible, but are restricted to single developer devices.
ETSI
8 ETSI TR 103 114 V1.1.1 (2013-10)
SM
4.2.3 Windows Phone 8
SM
Introduced by Microsoft Corporation , Windows Phone is the successor of the Windows Mobile family of operating
systems. Phone 8 is currently available in version 8.0 and implements smartphone capabilities on touchscreen devices.
SM
Different manufacturers make use of Windows Phone 8 as an operating system for their devices.
Some restrictions related to automated QoS testing apply whereas the API command set for information retrieval and
SM
test automation has been extended compared to Windows Phone 7 or previous versions of Windows Mobile.
This is also related to the requirement that all applications have to be checked in detail before being made available for
download and installation. Automation of QoS test applications has not been fully supported by this model yet.
SM
Applications are released via the Microsoft Windows Store. Direct access to a limited number of development
devices per developer license is granted as well.
SM
As a variant of Windows Phone 8 , the version Windows RT is available to support applications on tablets running on
SM
ARM processors. It uses the same OS kernel as Windows Phone 8 . However, only applications distributed via the
Windows Store can be deployed with Windows RT devices.
TM
4.2.4 Blackberry OS
TM TM
The company Research In Motion (RIM) published the operating system Blackberry OS which is used on many
messaging-oriented devices of this manufacturer. The newest released version X provides many useful API commands
to retrieve network related information.
TM TM
Applications based on Blackberry OS have to be checked and signed by RIM before they can be installed on
according devices. Each developer has to acquire an according set of signing keys in advance. ®
4.2.5 Symbian ®
The Symbian family of operating systems is available for many years and evolved over time. It turned into an open
operating system over time and can still be found on many devices in the field, nowadays mostly related to the low cost ®
price tier. Symbian has been one of the first available operating systems in the smartphone area.
The access to QoS test related information has always been complex and in many cases not officially supported.
Applications could be installed locally without any need to involve online instances of the vendors.
5 Limitations of Smartphones
To use smartphones as a basis for QoS testing comes with some limitations compared to usual host based systems.
These limitations are based on the physical device and on the allowed access to relevant information required for QoS
testing.
5.1 Device Limitations
On a physical level, mobile devices come along with limited resources. This covers mainly the areas:
• battery power;
• processing power;
• memory size;
• display size and resolution; and
• speed of the storage media (e.g. lik
...