ETSI TR 103 501 V1.1.1 (2018-10)

Speech and multimedia Transmission Quality (STQ); Guidelines for the Measurement of Data Throughput on Devices connected to Mobile Networks

DTR/STQ-00217m

General Information

• Status: Published
• Publication Date: 11-Oct-2018
• Technical Committee: STQ MOBILE - STQ Mobile
• Current Stage: 12 - Completion
• Due Date: 25-Oct-2018
• Completion Date: 12-Oct-2018

Standards Content (Sample)

TECHNICAL REPORT
Speech and multimedia Transmission Quality (STQ);
Guidelines for the Measurement of Data Throughput on
Devices connected to Mobile Networks

2 ETSI TR 103 501 V1.1.1 (2018-10)

Reference
DTR/STQ-00217m
Keywords
3G, data, GSM, network, QoS, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:
http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is the
print of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
https://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx
Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.

© ETSI 2018.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
TM TM
3GPP and LTE are trademarks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members. ®
GSM and the GSM logo are trademarks registered and owned by the GSM Association.
ETSI
3 ETSI TR 103 501 V1.1.1 (2018-10)
Contents
Intellectual Property Rights . 4
Foreword . 4
Modal verbs terminology . 4
Introduction . 5
1 Scope . 7
2 References . 7
2.1 Normative references . 7
2.2 Informative references . 7
3 Definitions of terms and abbreviations. 8
3.1 Terms . 8
3.2 Abbreviations . 8
4 Background . 9
5 Basics of throughput measurements . 9
6 Treating measurement and evaluation methodology as a unit . 10
7 System Boundaries . 11
8 Points of control and observation . 12
9 Measurement equipment considerations . 12
10 Measurement Modes . 13
10.1 Background . 13
10.1.1 General . 13
10.1.2 Fixed-size-method . 13
10.1.3 Fixed-time-method. 13
10.4 Selection of the most appropriate mode . 14
10.5 Practical examples . 15
11 Data Evaluation . 16
11.1 Basic considerations . 16
11.2 Test case parametrization and post processing aspects . 17
11.3 Using subsets of data points . 18
11.4 General aspects on reporting of throughput measurement results . 19
12 Considering equipment related effects . 20
13 Latency measurements . 20
14 Aggregation . 21
14.1 Overview . 21
14.2 Temporal or data-point aggregation . 21
14.3 Spatial aggregation . 22
15 Multi-socket measurements . 22
16 Comparability and reproducibility . 25
17 Summary and conclusion . 25
Annex A: Bibliography . 26
History . 27

ETSI
4 ETSI TR 103 501 V1.1.1 (2018-10)
Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys no
right to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.
Foreword
This Technical Report (TR) has been produced by ETSI Technical Committee Speech and multimedia Transmission
Quality (STQ).
Throughput, or data rate, is the single most important property of a packet data network. The definition of throughput -
transferred volume of data per unit of time - is essentially simple. However, there are many different methodologies
available to measure it. To select the most appropriate one for a given purpose, and to assess comparability of results,
requires thorough understanding of these methodologies. The present document addresses the measurement
methodologies that can be used from an end-user perspective, i.e. embedded in devices or in dedicated test equipments
connected to mobile networks.
While there is extensive coverage of IP layer centric methodology (such as ETSI EG 203 165 [i.1] ("Throughput
measurement Guideline"), the content of such ETSI Guide does not actually cover methodologies and aspects such as
application-level measurements. The present document takes also in consideration methods known as 'crowdsourcing',
which have gained considerable audience (and potentially relevance) in the last years but until the time of publication of
the present document have not been subject to extensive treatment in the framework of standardization work (however
there are activities under way, e.g. the E.MTSM work item in Question 12 of the ITU-T Study Group 12).
Likewise, the present document integrates multi-threaded measurements into a common methodological frame.
The present document has been written to provide a holistic, organized view of the entire measurement process, which
also includes elements such as definition of system under test and system boundaries, post processing of data and
relation between basic methodologies and their relation to intended targets of measurements.
Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to be
interpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.
ETSI
5 ETSI TR 103 501 V1.1.1 (2018-10)
Introduction
Throughput, or data rate, is the single most important characteristic of packet data networks. While its definition -
transferred data volume per unit of time - is simple, there is a wide range of possibilities how actual measurements can
be carried out. Consequently, it is hard to decide if results of different measurements are comparable of how results
from one type of measurement can be used to predict the outcome of another type of usage.
Also, there are entirely different views on performance characteristics of packet data networks. From a low-level
perspective (on the IP layer), a network transfers data packets and its performance is characterized by packet transfer
time, latency or delay and packet loss rate; also, the variations in time of these metrics, as well as less frequent events
such as packet re-ordering have an effect. On higher protocol layers such as TCP, there is no packet loss; lost packages
on lower layers translate into lower overall data rates. From an end-user, QoS or QoE perspective, the performance of a
network may again vary as the dynamics of a particular application interact with network characteristics and behaviour
of other components in sometimes complex ways. In addition, networks typically use resource and performance
optimization mechanisms which further increase the complexity of dynamic behaviour.
Mobile connectivity has become an important element of modern life. Both business and consumers have great interest
in knowledge about the performance of mobile networks and useful information about this performance is in high
demand. There are few actors which have the means to use professional measurement tools to obtain such information.
This is one of the reasons why in recent years, a substantial number of companies have emerged which develop and
distribute crowdsourcing tools - subsequently termed 'speed test apps' although this description is not entirely correct -
to measure mobile network performance. Due to promised or expected cost reduction, even network operators use and
rely on such tools today. In some countries, regulators operate crowdsourcing tools too.
From their mode of operation, these tools are considered to effectively measure network performance from an end
customer perspective, as the tools run on end-user smartphones. Use cases typically are http or ftp upload and download
and therefore results can be attributed to be QoS values and do not represent actual internet speed measurements as
understood by e.g. laboratory measurements or assessments in the regulatory context.
Strictly speaking, the scenarios used represent only a small fraction of actual end-user behaviour as pure upload and
download only plays a role in use cases such as app download or transfer of larger number of data as e.g. in transfer of
photos or videos (typically in e-mail or cloud storage contexts). Nevertheless, such tests play a significant role in the
public perception of mobile networks. Through the interplay between public media and PR of mobile network
operators, they have a substantial economic impact.
The basic requirement for any meaningful, professional measurement is repeatability. As long as the properties of the
network under test and of the test equipment or application stay the same, running the same test is expected to produce
the same results. Repeatability allows then comparison. But comparison can also be understood between measurement
tools or applications, and in such a case, it requires that the relevant procedures (parameters, set up) of the test are fully
documented. This is usually not the case with current 'speed test' applications, and even less so regarding full
interoperability, e.g. by open access to servers used as counterpart of throughput testing.
The present document provides a contribution to the evolution of network performance testing towards a professional
degree of transparency. This begins with a consistent framework of definitions and technical terms. The elements of the
testing process are then described within this context.
Apart from the obvious direct parameters of throughput testing, such as time windows or transferred data volumes, there
are numerous other elements which can have an impact on data values obtained. In this sense, methodology and
definition of metrics cannot be decoupled from each other. The process starts with selecting the boundaries to the
system under test, i.e. insertion or demarcation points. Next comes the way the system under test is accessed. For
instance, if the test is run over a radio access network using a mobile device such as a smartphone, the type and degree
of influence needs to be assessed. The type of stimulus is likewise important, such as the protocol type, the structure of
data traffic (e.g. TCP or UDP based), and the number of parallel connections. Depending on these selections, other
choices also become parameters for testing. An example would be to use some kind of real application to create a
particular type of traffic, versus using synthetically generated traffic.
The need for careful consideration is not limited to the generating side of measurement data. The way data is processed
may also have an impact on results and is therefore subject to documentation and transparency. This applies e.g. to rules
about which data to include in computation of results, and which to ignore or discard. For instance, a methodology may
require discarding a certain number or range of extreme values to reduce the volatility of results.
ETSI
6 ETSI TR 103 501 V1.1.1 (2018-10)
Beyond the direct uses of throughput measurement, one of the driving forces is the prospect of using respective data to
predict or infer QoS or QoE for a broader range of services. This is of course desirable from a commercial and practical
point of view, in order to reduce the complexity of testing as compared to running actual service test use cases. There is
no doubt that doing this is possible in principle, using different approaches such as a model-based ones or empirical
methods, i.e. a data driven mapping between results from both domains, can be used. It is beyond the scope of the
present document to discuss this topic in detail. It is however clear that a meaningful way to do so involves a large
degree of caution and professional care - e.g. in calibration and validation of methods. In any case, it will be necessary
to gauge the efficiency, data quality the overall effort of such approaches against direct service tests to obtain QoS or
QoE parameters by running actual use cases.

ETSI
7 ETSI TR 103 501 V1.1.1 (2018-10)
1 Scope
The present document provides a systematic overview of methods to measure throughput in mobile networks, with
special focus on measurements using a viewpoint at, or close to, application level. Also, it provides a holistic, integrated
view of the measurement process, which also includes a selection of methodologies according to intended goals of
measurement, and also covers post-processing and data aggregation aspects.
2 References
2.1 Normative references
Normative references are not applicable in the present document.
2.2 Informative references
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI EG 203 165 (V1.1.1): "Speech and multimedia Transmission Quality (STQ); Throughput
Measurement Guidelines".
[i.2] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
NOTE: The content of this document series has also been used (copied) in Recommendation ITU-T E.804.
[i.3] Recommendation ITU-T Q.3960: "Framework of Internet related performance measurements".
NOTE: Available at http://www.itu.int/rec/T-REC-Q.3960-201607-I/en.
[i.4] ETSI TR 102 678: "Speech and multimedia Transmission Quality (STQ); QoS Parameter
Measurements based on fixed Data Transfer Times".
[i.5] ETSI TS 138 521-3: "5G; NR; User Equipment (UE) conformance specification; Radio
transmission and reception; Part 3: Range 1 and Range 2 Interworking operation with other radios
(3GPP TS 38.521-3)".
[i.6] ETSI TS 138 101-3: "5G; NR; User Equipment (UE) radio transmission and reception;
Part 3: Range 1 and Range 2 Interworking operation with other radios (3GPP TS 38.101-3)".
[i.7] Recommendations ITU-T Y.154x series: "Quality of service and network performance".
[i.8] Recommendation ITU-T Y.1545.1: "Framework for monitoring the quality of service of IP
network services".
[i.9] IETF RFC 7398: "A Reference Path and Measurement Points for Large-Scale Measurement of
Broadband Performance".
ETSI
8 ETSI TR 103 501 V1.1.1 (2018-10)
3 Definitions of terms and abbreviations
3.1 Terms
For the purposes of the present document, the following terms apply:
data path: sequence of entities or elements (physical/virtual) which a data packet transferred between endpoints
traverses
endpoint A: local entity in an end to end test scenario (typically, the actual test system)
endpoint B: remote party in an end to end test scenario (in packet data tests, often a server connected to the public
internet; can also be a CDN in case of live public content
fixed-size method: throughput measurement method where a fixed amount of data is transferred and the time for
transfer is recorded to calculate a throughput value
fixed-time method: throughput measurement method where data transfer is performed for a fixed period of time, and
the amount of data is recorded to calculate a throughput value
3.2 Abbreviations
For the purposes of the present document, the following abbreviations apply:
API Application Programming Interface
CDN Content Delivery Network
DL DownLoad
FTP File Transfer Protocol
HTTP Hyper Text Transfer Protocol
HTTPS HTTP Secure
HW HardWare
ICMP Internet Control Message Protocol
IP Internet Protocol
IT Information Technology
ITU-T International Telecommunication Union - Telecommunication
MDR Mean Data Rate
NuT Network under Test
OTT Over The Top
PoO Point of Observation
PR Public Relations
QoE Quality of Experience
QoS Quality of Service
QUIC Quick UDP Internet Connections
RAT Radio Access Technology
RTT Round Trip Time
SI International System of units (Système international d'unités)
TBKPI Time Based Key Performance Indicator
TCP Transport Control Protocol
TP ThroughPut
UDP User Datagram Protocol
UL UpLoad
ETSI
9 ETSI TR 103 501 V1.1.1 (2018-10)
4 Background
There are standards documents that deal, in part quite extensively, with pure throughput measurement at IP level, or
with measurements where throughput or round trip time is part of a larger metric for a given use case or service. The
main reason for considering round-trop time together with throughput is that TCP throughput is dependent on both the
minimum capacity of all path segments, and the round-trip time (because of the feedback loop in TCP flow-control, this
would apply to QUIC or any retransmission protocol in general).
The common factor in these standards is that they focus strongly or exclusively on methods based on events of lower
protocol levels (typically, the IP or TCP plane). However, the reality at the time of publication of the present document
is that there are various tools available - which sometimes appear to be quasi-standards making use events from higher
layers - for which there are good reasons, explored further in the course of the present document.
NOTE 1: The term application plane is used here as a synonym for Points of Observation (PoO) above the IP layer.
Actually, events used in throughput measurement may come from any layer between the API for basic
data transfer (e.g. the operating system's socket API) and user-interface indicators in case real smartphone
apps are used. According to the principle laid out in ETSI TS 102 250 [i.2], mixing events from different
PoO should be avoided whenever reasonably possible.
NOTE 2: Measurement on radio carrier are also possible, but outside of the scope of the present document. For
details on this topic, one can refer to ETSI TS 138 521-3 [i.5] and ETSI TS 138 101-3 [i.6].
At the time of writing, there appears to be no document which provides a comprehensive, practically oriented overview
of all aspects of the entirety of packet-data network performance measurements, which goes far beyond core
measurement methodology and integrates aspects of system boundaries and data aggregation.
While IP-level methods are also mentioned in the present document for completeness, its main focus is on the
application plane, and reference is made to documents which treat IP-level measurements extensively, such as ETSI
EG 203 165 [i.1] or Recommendation ITU-Ts of the Y.154x series [i.7].
Application-level measurements can be run on practically all devices while low-level data is typically only available if
devices are modified, granting apps full system access. However, such modifications, usually called rooting, render a
device potentially unsafe (typically, this process removes some security features). Even if users would accept that,
which is considered very unlikely, the process of rooting is not easy to perform, can lead to permanent damage of
devices of something goes wrong, and voids device warranty. Therefore, it is safe to assume that any larger distribution
of measurement apps, and in particular crowdsourcing, will have to be based on application-level measurement
methods.
This does not mean that application-level methods provide an entirely different point of view. It is assumed that there
are clear, deterministic relations between the layers of packet data transfer, so in principle a direct relation to QoS
metrics based on low-level events can be established. In this respect, application-level methods provide access to a
much broader range of information sources.
Remark: Relations between layers are testable so respective validation can be done when needed.
One aspect has to be taken into account, namely the potential higher degree of device dependency, which requires
additional professional care at the level of test design and conduction. Also, application-level measurements may
produce less diagnostic depth than measurements on IP level. On the bottom line, the method of choice will be selected
based on the type and depth of required information and the relative cost and effort of obtaining it.
5 Basics of throughput measurements
Throughput is defined as transferred data volume per time, or:

TP =
Where the unit of TP is typically kbit/s.
ETSI
10 ETSI TR 103 501 V1.1.1 (2018-10)
NOTE: As the amount of transferred data (data volume) is often given in kByte or Mbyte, some caution is
3 6
indicated. The SI defines prefix k stands as 10 and prefix M as 10 . In the IT world and even in parts of
related standardization literature, the prefixes k and M are used for factors based on powers of 2. The
prefix k equals 2 (1 024) and M equals 1 024 × 1 024. Obviously, even a single misuse of the k prefix
already has a potential error of 2,4 %. This error may multiply for consecutive wrong usages. For
technical purposes, the safest way is to not use prefixes at all, but byte values. When this cannot be
applied, great care is advisable throughout the documentation chain.
Values for data volume and duration are easy to obtain. However, there are multiple sources for each of them (taken at
different points of observation, PoO) and values can only be compared if the PoO are the same.
Practically, events can be closely linked i.e. by a (demonstrable) systematic deterministic relationship (e.g. a function-
call chain) so even if the formal PoO is not the same, it can be treated as equal. This applies even to a case where this
chain involves a fixed time differential which then would have to be taken into account.
The PoO aspect is not just a formal one, it is also related to the QoS level. If, for instance, a given data transmission
path has a non-negligible packet loss rate, the low-level data transmission activity or rate may be high but with no
practical use from a user's perspective. Also, transmission protocols may have different amounts of overhead that lead
to different user-perception throughput values for the same low-level data rate.
This aspect has a special dimension when using API functions of certain device classes, e.g. the traffic counters of
smartphones. When using such data sources, it is necessary to know the level at which these traffic counters are actually
counting. At least, the methodology in the context of repeatability of measurement will need to include an element of
continuity assurance.
Likewise, the duration needs a proper definition. It is given by two points in time. The throughput of a network under
test is not constant over time, in particular at the beginning of a transmission. Therefore, the choice of specific events or
otherwise defined points in time will have an impact on the throughput value obtained from a measurement.
In the case of upload data rates, it is important to understand if the data regarded as transmitted is just successfully
handed over to the next stage in the chain, or if it is actually transferred across the network under test.
In case of benchmarking measurements, as all channels are using the same architecture and methodology, all aspects
mentioned above are less critical. Usually, they do not affect the ranking order of candidates. In case of single-channel
measurements, results are usually compared to those measured with other systems or taken at an earlier point in time.
Here, it is important to record the used parameters, and to properly understand eventual effects.
Calibration or connection measurements, as used in other fields of engineering, can also be a useful method to ensure
consistency. Such measurements would involve measurements on the same system under test using the previous and the
new tool or methodology, and producing a statistically relevant number of samples in order to understand eventual
differences in output values.
As a summary, throughput measurements are not intrinsically complex. In order to make sure that the results are correct
and consistent, there are, however, some principles of proper craftsmanship to be observed.
6 Treating measurement and evaluation methodology
as a unit
Practical experience shows that measurement and evaluation methods cannot be treated separately. For a full
understanding of applicable evaluation methods and interpretation of results, the method used needs to be known. The
single most important example is the selection of file size (or time window) in throughput measurements.
A certain minimum size is necessary to avoid ramp-up and quantization effects. With typical TCP window sizes in the
range of 1 Mbit and beyond, and data rates in the order of 100 Mbit/s, a transfer of less than 3 Mbytes to 6 Mbytes will
not produce any meaningful results.
Typically, throughput figures obtained from measurements with a particular set of parameters (protocol type
e.g. ftp/http; file size) need to be extrapolated to obtain predictions for other parameters, too. It is important to
understand how reliable these extrapolations are, considering that a network under test may exhibit a different
behaviour if parameters are different. The term prediction horizon will be used to elucidate this situation.
ETSI
11 ETSI TR 103 501 V1.1.1 (2018-10)
For instance, a resource optimization mechanism may decide that FTP traffic has a lower priority than HTTP. A
throughput measurement using an FTP download scenario would then not be able to deliver accurate predictions for
usage of HTTP download. Likewise, there may be mechanisms such as fair use policies which reduce the effective data
rate after a certain amount of data has been transferred, or strategies which do the opposite, i.e. increasing throughput
after some time to deliver a better QoE for large-volume transfers.
With upcoming 5G, and the feature of network slicing, it is very likely that the prediction horizon of measurements will
be reduced further.
The exact function of current resource optimization mechanisms is not publicly known (and would be different for each
network anyway). Also, network behaviour can change. Therefore, it is highly advisable to apply reasonable
precautions, e.g. pre-campaign validation runs or result monitoring to make sure that assumptions crucial to the success
of a measurement are valid. The safest way is in any case to make sure that test case parameters are sufficiently close to
the actual use cases for which the measurement results are intended to be used.
7 System Boundaries
At the core of every network performance measurement, a clear definition of the object under test is required.
Figure 7.1 shows the principal components.

Figure 7.1: Schematic of test environment and network under test
The following practical terms are introduced:
• Endpoints: The principal origin and destination element/device of data packets used for performance testing.
In some contexts, the terms 'A Party' and 'B Party' are also used.
• Data Path: This is the route for the data packets which are transferred between the endpoints.
Figure 7.1 describes a functional architecture, with the main purpose of defining the boundaries between the Network
Under test (NUT) and the surrounding elements.
In an actual test setup, some of the elements may be combined in one physical unit (as in the case of a smartphone
where the A Party test system and the access device are integrated). Likewise, a functional unit may be distributed in
physical entities, or the B Party might not be a single data server, but a content delivery network made up of thousands
of physical or virtual servers in different locations, with some load- or latency-optimizing logic on top.
The 'Interconnection Domain' element has a special meaning. Even if the purpose of a measurement is to characterize a
particular network, its principal connections, at the exact edge of the network to the outer world, may not be accessible.
In this case, suitable access or interconnection points are used that constitute part of the data path. This is more than a
technical issue. Network performance metrics are often a part of regulation. A network operator can only take
responsibility for properties which are under his control. Therefore, the definition of system boundaries is a sensitive
issue, which requires great care.
For further reading on this topic, it is recommended to refer to Recommendations ITU-T Q.3960 [i.3],
ITU-T Y.1545.1 [i.8] and IETF RFC 7398 [i.9].
For QoS measurements, the endpoint B is often some kind of public server, in particular for web browsing tests which
at the time of writing use a mix of some static reference pages, and some live web pages. For performance
measurements, a similar architecture with some general counterpart servers is conceivable. Such architectures require,
however, some kind of additional validation structures or processes to ensure that third-party inflicted effects do not
compromise measurements. Therefore, performance measurements typically use dedicated servers so that both
endpoints of performance measurements are under the control of the entity performing the tests.
ETSI
12 ETSI TR 103 501 V1.1.1 (2018-10)
8 Points of control and observation
Any performance test requires a minimum set of information, which is typically comprised of events on some protocol
layer. On lower layers (e.g. IP), the sources of such events are typically traces, i.e. software functions delivering
information on packet level. For application-level testing, activities are started, and events are observed, using primary
functionality provided by the operating system (API, application programming interfaces), or corresponding interfaces
of other software packages on top of such APIs.
Controlling and observing activities can use the same or different interfaces. For instance, an active throughput
measurement needs to send some data, which is usually on the API level or higher. For IP level based measurement, this
activity is then monitored at another place in the data chain. For application-level testing, the location for control and
observation can be the same.
A special case (and one of the weak points of IP-level testing) is a situation where data transfer is encrypted (e.g. using
HTTPS instead of HTTP) or uses a proprietary protocol (in case of testing using OTT functionality). This renders
low-level events completely inaccessible, or, at best, imposes additional processing requirements such as some kind of
pattern recognition to link low-level to application-level activities. Such features are also often device-dependent.
9 Measurement equipment considerations
There are many ways to realize measurement equipment. As long as some basic common sense requirements are
fulfilled, there is no better or worse equipment. Even if there are more or less precise instruments (ranging from a
microsecond instead of millisecond time resolution to extensive hardware-heavy solutions such as temperature
stabilized modems or smartphones), their practical use needs to be evaluated against the practical advantage on a cost to
value axis, on the background that in a live network properties fluctuate on significant scales.
There are, however, some basic properties, which need to be fulfilled in any case to make a measurement valid. The
most important of these properties is that the measurement system will not have uncontrolled effects on the
measurement values.
With respect to throughput measurement, the strict version of this requirement would be that the system may not be the
bottleneck in a measurement, i.e. limit by itself some of the values it measures. This demand, however, may be too strict
in an efficiency-oriented frame. For example, it may make sense to use bandwidth-limited servers or systems. These
would of course not be adequate for measuring the peak data rate a NUT can achieve. It may still be a good choice if the
primary interest of a measurement is to check if a network can provide a given minimum data rate. The advantage of a
data rate limitation, on the other hand, is that the amount of data transferred is kept in check. The practical value, e.g. in
cases where a high speed data volume limit exists for standard SIMs, can be higher than the potential drawback of not
knowing the peak data rate of the NUT.
Well-understood restrictions may also be related to certain operation parameter ranges. For instance, running IP Trace
imposes - at contemporary data rates of several 100 Mbit/s - considerable extra load to a system in order to process or
even just store a huge amount of extra data volume. Assuming that basic measurement system performance is adequate,
there still may be other effects, such as a drift in time stamps of captured packets for transactions longer than a given
duration, due to a data queue which runs up when the system's mass-storage limits the speed.
The essential requirement is therefore that the relevant properties of the testing system will be well characterized. It will
be possible to know if a particular measurement value is a property of the NUT, or somehow influenced by the system.
A special case to be considered, in the context of comparability of measurements, is continuity. Realistically, any
measurement system needs to be kept up to date, which involves hardware or software updates or upgrades. For
instance, given the speed of evolution in RAT, a given modem or smartphone will have to be replaced, sooner or later,
with a newer and usually more capable model. Also, operating system updates will have to be performed, if for no other
reason than to keep up with typical user perspective. Another important element of consideration for comparability of
measurement is the possibility to run it across several radio access technologies (3G, 4G, WiFi, etc.), including
situations where hand overs between these technologies occur. If comparability in such situations cannot be ascertained,
then measurement systems needs to be able to block RAT, frequency, and in any case to report under which technology
a given measurement has been performed.
ETSI
13 ETSI TR 103 501 V1.1.1 (2018-10)
It is desirable that a system upgraded in one of these ways delivers exactly the same values as before. Practical
experience shows, however, that there are cases where the effort to achieve this goal is unreasonably high. A solution
can be using connecting measurements, a technique which is used in other fields of metrology. This technique is useful
for both validation of a new system based on a previously existing one, or to establish mapping relations between
results of the previous and the new system.
10 Measurement Modes
10.1 Background
10.1.1 General
A throughput measurement can be done by transferring a fixed amount of data (fixed-size method) and measuring the
time required, or by transferring data for a fixed amount of time (fixed-time method), and recording the amount of data
transferred.
NOTE: Several names or terms have been used for the fixed-time and fixed-size methods. The fixed-size method
has been called "best effort method" in ETSI EG 203 165 [i.1]. For the fixed-time methods, terms such as
"windowed (ETSI EG 203 165)", "Fixed Data Transfer Time QoS (FDTT-QoS)" (ETSI
TR 102 678 [i.4]), "time based", or TBKPI are being used - this list is not even intended to be complete.
The choice of the name pair fixed size/fixed time used in the present document is intended to clean up this
situation with a consistent and self-explanatory pair of terms.
Both methods have their specific properties, benefits, and drawbacks.
10.1.2 Fixed-size-method
The fixed-size method is, on first glance, more user experience related: the typical use case is a transfer of some kind of
data file. For measurement purposes, it has, however, the disadvantage that it introduces a large variation of time per
transaction. The spread of data rates in the field current ranges from less than 100 kbit/s for 2G up to 300 Mbit/s for 4G.
A long transaction time is undesirable in several ways. In benchmarking, it inevitably leads to a very fast
de-synchronization of transaction types in mixed scenarios. At best, with pure upload or download scenarios, the effect
is a large difference in sample count which is also undesirable. Last but not least, in drive test scenarios long transaction
times lead to small sample densities or low spatial resolution.
To end up with a reasonable maximum transfer time for 2G, file sizes would have to be so small that the peak data rate
in 4G would not be visible due to ramp-up. The only way left to at least dampen this effect is a time-out which cuts off
a transaction after a given maximum time. Applying the usual validity rules for samples in QoS measurement, the
downside of this solution is that affected transactions would be removed entirely from eval
...