ETSI EG 203 165 V1.1.1 (2012-04)

Speech and multimedia Transmission Quality (STQ); Throughput Measurement Guidelines

DEG/STQ-00162m

Kakovost prenosa govora in večpredstavnih vsebin (STQ) - Smernice za merjenje pretočnih zmogljivosti

General Information

• Status: Published
• Publication Date: 26-Apr-2012
• Technical Committee: STQ MOBILE - STQ Mobile
• Current Stage: 12 - Completion
• Due Date: 08-May-2012
• Completion Date: 27-Apr-2012
• Ref Project: kSIST-V ETSI/prEG 203 165 V1.1.1:2012 - Speech and multimedia Transmission Quality (STQ) - Throughput Measurement Guidelines

Standards Content (Sample)

Final draft ETSI EG 203 165 V1.1.1 (2012-02)

ETSI Guide
Speech and multimedia Transmission Quality (STQ);
Throughput Measurement Guidelines

2 Final draft ETSI EG 203 165 V1.1.1 (2012-02)

Reference
DEG/STQ-00162m
Keywords
3G, GSM, network, QoS, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2012.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
Contents
Intellectual Property Rights . 5
Foreword . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions, symbols and abbreviations . 7
3.1 Definitions . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 General throughput measurement aspects . 8
4.1 Purpose of the measurement . 8
4.2 Throughput equation . 8
4.3 Throughput on different layers . 9
4.4 Throughput at different reference points . 10
4.5 Active and passive measurement . 11
4.6 Load generation for active measurement of IP throughput . 11
4.7 "Best effort" versus "windowed" approach . 12
4.7.1 "Best effort" approach. 12
4.7.2 "Windowed" approach . 13
4.7.3 Bias effect for "best effort" and "windowed" approaches . 13
4.7.4 First example . 13
4.7.5 Conclusions from first example . 14
4.7.6 Second example . 14
4.7.7 Conclusions from second example . 15
4.7.8 Combining "best effort" and "windowed" approach . 15
4.8 Upper bounds for TCP throughput measurements . 15
5 Measurement environment aspects . 15
5.1 Client/Server hardware . 15
5.2 Operating System . 16
5.3 Performance Enhancement Proxies . 16
5.4 Shared medium . 17
5.5 All traffic versus filtered traffic . 17
5.5.1 User traffic . 17
5.5.2 Multithreaded applications . 17
5.5.3 Aggregate traffic . 18
6 Campaign planning and evaluation . 18
6.1 Mean user data rate versus mean transfer time measurements . 19
6.1.1 Example . 19
6.1.2 Conclusion . 21
6.2 Throughput calculation based on time or traffic . 21
6.2.1 Non-sampled averaging . 22
6.2.2 Sampled averaging . 22
6.3 Busy hours or peak-off . 22
6.4 Working days or weekends . 22
6.5 Locations . 22
6.5.1 Mobility aspects . 22
6.5.2 Area categories . 22
6.6 Calculating sample mean . 22
7 Throughput measurement checklists . 23
ETSI
4 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
Annex A: Analysing IP traces under different aspects . 27
A.1 Layer-based analysis . 27
A.2 User-based analysis . 27
A.3 Volume-based analysis . 27
A.4 Time-based analysis . 28
A.5 Examples . 28
History . 29

ETSI
5 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This final draft ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia
Transmission Quality (STQ), and is now submitted for the ETSI standards Membership Approval Procedure.
Introduction
The main purpose of the present document is to help the reader understand and differentiate between various throughput
definitions and calculation methods described in the TS 102 250 series and technical reports produced by STQ.
This guide describes the different aspects (e.g. protocol-specific, measurement-environmental, statistical) which should
be considered during planning, execution and evaluation of throughput measurements in order to avoid the major
problems which can occur.
TS 102 250-2 [i.2] standardizes throughput QoS parameters for popular IP based services used in mobile networks from
the user's point of view. Based on these definitions TS 102 250-7 [i.6] defines a model for network quality. Finally,
TR 102 678 [i.8] introduces a new method of throughput calculation based on fixed data transfer times.
ETSI
6 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
1 Scope
The present document focuses on aspects of throughput measurements and their evaluation, providing different
approaches. It contains factors, guidelines and background information that should be considered during selection,
planning, execution and evaluation of throughput measurements.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-1: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 1: Assessment of Quality of Service".
[i.2] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.3] ETSI TS 102 250-3: "Speech and Multimedia Transmission and Quality (STQ); QoS aspects for
popular services in mobile networks; Part 3: Typical procedures for Quality of Service
measurement equipment".
[i.4] ETSI TS 102 250-4: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 4: Requirements for Quality of Service measurement
equipment".
[i.5] ETSI TS 102 250-6: "Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects
for popular services in GSM and 3G networks; Part 6: Post processing and statistical methods".
[i.6] ETSI TS 102 250-7: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in GSM and 3G networks; Part 7: Network based Quality of Service
measurements".
[i.7] ETSI TR 102 607: "Speech Processing, Transmission and Quality Aspects (STQ); TCP IP Stack
Parameter Settings for Microsoft Windows XP and Microsoft Windows Vista; Comparison and
Recommendations".
[i.8] ETSI TR 102 678: "Speech and multimedia Transmission Quality (STQ); QoS Parameter
Measurements based on fixed Data Transfer Times".
ETSI
7 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
[i.9] ETSI TR 102 807: "Speech and multimedia Transmission Quality (STQ); Process description for
the transaction view model".
[i.10] ITU-T Recommendation X.290: "OSI conformance testing methodology and framework for
protocol Recommendations for ITU-T applications - General concepts".
[i.11] IETF RFC 793: "Transmission Control Protocol".
[i.12] IETF RFC1323: "TCP Extensions for High Performance".
[i.13] IETF RFC 3481: "TCP over Second (2.5G) and Third (3G) Generation Wireless Networks".
[i.14] M. Mathis, J. Semske, J. Mahdavi, and T. Ott: "The macroscopic behavior of the TCP congestion
avoidance algorithm." Computer Communication Review, 27(3), July 1997.
3 Definitions, symbols and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
application: software, using a particular service for providing related functionality to the user
Point of Control and Observation (PCO): point within a testing environment where the occurrence of test events is to
be controlled and observed
NOTE: A PCO is identified by a) a reference point or interface and b) a service access point (SAP) at the
specified reference point or interface, indicating unambiguously where (usually in a protocol stack)
events are observed (or other measurements are made). See ITU-T Recommendation X.290 [i.10].
service: capability of a specific layer and the layers beneath to provide a set of functions to higher layers
NOTE: One example of the higher layers is the application layer.
transmission capacity: maximum achievable throughput, which is determined by the physical characteristics of the
transmission media
NOTE: Examples of the physical characteristics of the transmission media are transmission capacity, applied
modulation and coding scheme.
3.2 Symbols
For the purposes of the present document, the following symbols apply:
Δt Predefined measurement time period
d
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
CDF Cumulative Distribution Function
CPU Central Processing Unit
FDTT Fixed Data Transfer Time
FTP File Transfer Protocol
FTTX Fiber To The X (of any type)
GGSN Gateway GPRS Support Node
HDD Hard Disk Drive
HSPA High Speed Packet Access
HTTP HyperText Transfer Protocol
IP Internet Protocol
ETSI
8 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
ISO International Organisation for Standardization
MSS Maximum Segment Size
NAT Network Address Translation
NP Network Performance
OSI Open System Interconnection
PCO Point of Control and Observation
PEP Performance Enhancement Proxy
QoS Quality of Service
RAM Random Access Memory
RTP Real Time Protocol
RTT Round-Trip Time
SDP Session Description Protocol
SIM Subscriber Identity Module
SYN TCP synchronise flag
TCP Transmission Control Protocol
TCP/IP Transmission Control Protocol / Internet Protocol
UDP User Datagram Protocol
UMTS Universal Mobile Telecommunications System
USB Universal Serial Bus
4 General throughput measurement aspects
When measuring throughput, certain protocol-specific aspects always have an impact on measurement results. Thus, the
purpose of the measurement and the respective measurement methodology has to be taken into account when evaluating
and comparing measurement results. The specific protocol used to perform a measurement may have an impact on
measurement results. In particular, throughput measurement results obtained using one protocol cannot generally be
assumed to be transferrable to other protocols. For example, performing application layer measurements in two different
mobile networks using both FTP and HTTP can result in a situation where one network produces better results for FTP
measurements while the other one produces better HTTP results.
4.1 Purpose of the measurement
The three main parameters to characterise the actual performance of an IP network are delay, packet loss and
transmission capacity.
The main cause why throughput is measured is to determine the achievable portion of the transmission capacity, being
influenced by delay and packet loss, for different services. From a user's point of view the throughput is a key
performance measure for a dedicated service perceived by the user on application level.
There is a notable difference between network throughput and application layer throughput which allows statements
about Network Performance (NP) and Quality of Service (QoS), respectively. A proper disambiguation of the different
perspectives can be found in clauses 5.1 to 5.3 of TS 102 250-1 [i.1].
4.2 Throughput equation
Throughput can be calculated using the following equation.

The achievable IP transmission capacity is a highly variable stochastic parameter that varies in both small and large
timescales. Every aspect of the throughput value has to be carefully listed; otherwise it can lead to misunderstandings.
ETSI
9 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
4.3 Throughput on different layers
During packet transmission many protocols are involved. On every protocol stack layer the throughput metric can be
interpreted. While from a user's point of view only the highest protocol stack layer is important, from a network
operator's point of view the throughput on every protocol stack layer has its own meaning.
Depending on the used layer different PCOs apply and thus, the respective measurement results need to be treated
accordingly. The concept of PCOs is explained in clause 7.1 of TS 102 250-1 [i.1] and in clause 5.1 of TR 102 807 [i.9].
For network throughput measurements, different measurement environments and also different tools are needed to be
compare to application layer measurements, since different protocol layers are involved accessible at their own
reference points. Thus, the protocol layer has to be chosen and protocol-specific aspects like e.g. stationary state for
TCP have to be taken into account. For performance assessments of different services with respect to these network
characteristics, corresponding application traffic has to be generated and measured on a network with "live" overall
traffic load. Network throughput measurements and application layer throughput measurements cannot be mixed, even
though both use service protocols such as FTP or HTTP.

Figure 1: Idealistic view: ISO OSI Reference Model
The ISO OSI Reference Model is shown in Figure 1. The relation between the different protocol layer throughputs is
typically the following:
NOTE 1: Due to headers on higher layers the lower layer throughput can normally be assumed to be greater than
higher layer throughput, i.e. ">" and not "≥".
Since every newly introduced layer increases the overhead, the amount of transferred data needed just for the
communication increases, thus decreasing protocol efficiency:

NOTE 2: As stated above, higher layers include headers which add traffic and additional data to the transferred data
which is not directly available or usable for the user.
ETSI
10 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
NOTE 3: In the real world there are exceptions from the guidelines stated above as described in Figure 2 (real
world example). Certain protocols, e.g. the Packet Data Convergence Protocol, are designed to increase
protocol efficiency as compared to higher layers by performing a header compression with the goal of
efficiently using scarce radio resources. Another example is the use of optimizer/compression software
that can modify user data in order to increase the perceived throughput. In this case one can also say that
lower layer throughput is bigger than higher layer throughput.

NOTE: Red circles indicate examples of header/payload compression.

Figure 2: Real world example: UMTS Protocol Stack (simplified)
with internet applications and possible optimization
Therefore, the throughput performance of an end-to-end communication is usually different from a user's point of view
compared to a network's point of view. In IP environments, upper layer protocols are executed in the end points
(e.g. terminals) while the network only performs a transport function involving only lower layer protocols. The end user
who is affected by the whole end-to-end path – including terminals - has to deal with the full protocol stack while the
network only deals with a subsection of the path involving only a few protocol layers.
Reporting just the measurement results without providing information about the layer involved and the protocol used
and the purpose of the measurement, e.g. network or application measurement, can lead to misinterpretation.
Furthermore, different service protocols use the underlying transport protocols differently and should thus be reported,
as well. For further details, please refer to clause 5.5.2.
In certain testing environments a service will not or cannot be evaluated by trigger points defined on just one layer. Start
and end trigger points can for example be on different layers or need to be mapped to other layers. For example, when
using a Performance Enhancement Proxy (PEP) the mapping of trigger points may refer to a point between the network
and the application layer, most commonly the transport layer. A detailed look at the influences of PEPs can be found in
clause 4.2.1 of TS 102 250-2 [i.2].
4.4 Throughput at different reference points
From a user's perspective the achievable throughput on the end-user side is important. This metric can be quite different
if it is applied at any other point of the network.
In case of TCP for instance, a connection terminates in two dedicated points from the user's point of view, e.g. the PC
running the web browser that is accessing a web server on a remote machine. From the network point of view, many
intermediate connections and therefore many more endpoints can be found, e.g. routers or firewalls. The different
properties of these TCP connections (traffic, retransmission rate) will result in different throughputs.
ETSI
11 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
Performing network throughput measurements against these endpoints (destination hosts) can help finding bottlenecks
in the network or to compare different IP aggregation points along the service line. As an example a feasible
measurement scenario for testing different GGSN performance can be to measure the achievable TCP transmission
capacity against a FTP server located at the GGSN's Gi interface. Another measurement scenario would be to test the
performance of firewalls or NAT or L3-L7 deep packet inspection tools along the service line by placing an FTP server
before and after these equipments and comparing the measurement results. Measurement results for different destination
hosts on different logical places along the network should not be aggregated.
4.5 Active and passive measurement
An active measurement generates specific traffic on the network under test and is commonly used on access level. A
passive measurement captures "live" traffic at dedicated points in the network. This type of measurement is usually used
in the core network.
It is difficult to calculate the achievable throughput per user from the captured "live" traffic in the core network. For
such passive measurements it is e.g. not known if the user did not want to or could not generate more traffic on the
access side due to technical limitations of the client applications. Furthermore, the user behaviour is not predictable, but
has direct impact on the calculated throughput.
When using an active measurement to determine the core link capacity in terms of the maximum achievable throughput,
it should be considered that there usually is additional user and/or signalling traffic on the core link. In such a scenario,
the active measurement will measure the throughput that is achievable with the remaining (free) network capacities not
already allocated to other resources/traffic.
Commonly, active throughput measurements are performed on access network side generating traffic in order to
saturate the available transmission capacity. Passive throughput measurements, on the other hand, are more common in
the core network.
4.6 Load generation for active measurement of IP throughput
There are several ways to evaluate the IP network throughput from an end-user's perspective. Most commonly the IP
network throughput is measured based on UDP or TCP since it cannot be measured directly on layer 3. In IP networks
the respective traffic generation is usually done using TCP, because the majority of applications nowadays use this
protocol to transfer data. UDP can also be used for throughput measurements, since the measured UDP throughput
represents the upper bounds for TCP throughput in case the IP packets are in the same QoS class.
TCP was designed to adapt to the actual IP transmission capacity and to avoid congestion. To this end TCP provides
mechanisms such as e.g. slow start, fast retransmit and fast recovery. In addition TCP constantly monitors the actual
RTT and packet loss rates and uses those values to optimize the transmission capacity utilisation. For the setup and the
evaluation of the measurement many protocol specific aspects should be considered with respect to TCP, such as e.g.:
• TCP/IP stack parameters;
• stationary state of TCP;
• RTT;
• packet loss.
The aspects listed above influence the achievable throughput. For further information concerning the TCP/IP stack
parameters, please refer to TR 102 607 [i.7].
When speaking about throughput a stationary state is assumed, i.e. the TCP congestion control mechanisms had enough
time to adapt to the actual capacity of the network.
Usually the stationary state throughput itself reflects the throughput TCP can reach in the long run, with the time it takes
to reach it also being important. To assess how fast users can access network resources, measurements with small files
are recommended due to the fact that the TCP slow-start has a stronger effect compared to large files.
ETSI
12 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
A common way to generate TCP traffic is by using FTP or HTTP. In case of the evaluation of the achievable
throughput the focus is not on the service evaluation, i.e. the service is used in order to generate traffic. However, it has
to be taken into account that other network specific aspects can influence the achievable throughput for different
services and thus the achievable IP throughput, such as e.g.:
• protocol-dependent routing;
• intermediate nodes, e.g.:
- PEP;
- NAT;
- firewalls.
When the traffic generator, e.g. some FTP server, is able to send out data packets with a higher rate than the network
can transmit the difference between the different application layer protocols is negligible assuming that any Layer 7
analyser running in the network analyses the packets at the same speed and TCP has achieved a stationary state. E.g. all
IP services using TCP as a transport service should provide nearly the same results when TCP has achieved a stationary
state. This might not necessarily be true when QoS classes are used on the Internet and in access networks (mobile,
xDSL, FTTX) assigning services to different service classes, for which specific service profiles might be defined. With
respect to this, the protocol used to measure the achievable IP throughput for a specific service class has to be chosen
carefully, e.g. FTP for background traffic, HTTP for interactive traffic and RTP for real-time traffic.
In case of UDP the measurement is more straightforward as UDP does not use any throughput control mechanisms like
TCP does: packets are sent by the sender without any regard of the capabilities of the intermediate network or the
receiver, i.e. the sender will not adapt to the actual transmission capacity and the measurement only counts the received
bytes.
From a test methodology's point of view the packet send rate should be chosen carefully and has to be higher than the
expected achievable transmission capacity of the network under test. Also, the expected packet loss should be
considered.
Although a UDP measurement provides a very good estimation for IP throughput if the measurement is well designed,
it does not provide any usable estimation for TCP throughput. The reason behind this is that no RTTs and packet loss
rates are measured and no traffic and congestion control functions will be taken into account. In addition network
elements such as traffic shaper or profiler usually only influence TCP traffic by dropping packets, but not UDP traffic.
4.7 "Best effort" versus "windowed" approach
In the following clauses the "best effort" and "windowed" approaches are introduced and compared using two
examples.
4.7.1 "Best effort" approach
The "best effort" approach is a best-practice method to achieve a maximum of measurement samples for a fixed test file
size within a given time. Measurement locations/areas with higher achievable throughput contribute more measurement
samples than those with lower achievable throughput. The mean value of all measured speed samples would be biased
by this effect.
Using the "best effort" approach as many data transfers as possible are measured within a given time. The number of
transfers depends on data transfer speed whereas the transferred amount of data per transfer is constant (not considering
retransmissions).
Figure 3: "Best effort" approach
Figure 3 illustrates the best effort approach. A number of subsequent transfers is measured but not all transfers last for
the same duration.
ETSI
13 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
4.7.2 "Windowed" approach
Using the "windowed" approach, data transfers are executed within measurement windows of predetermined length.
The number of transfers is constant and the transferred amount of data per transfer depends on the data transmission
ratio.
Figure 4: "Windowed" approach
Figure 4 illustrates the windowed approach. Here it can be seen that all transfers last for the same duration, i.e. the
configured window size.
The "windowed" approach is best suited to achieve a more regular kind of sampling for network performance. As the
sampling is regular in the time domain one could approximately achieve a regular geographical sampling by doing tests,
e.g. as drive test with low but constant velocity. The mean value of all measured transfer speeds would better reflect the
average experienced speed under the assumption that users use the network at random locations.
As in reality network usage is not random or evenly distributed onto the area a geographic traffic distribution would
have to be considered to achieve a mean speed value even better reflecting user experience.
With respect to the findings above and especially in order to fulfil the common request for many QoS measurements to
have a limited and regular runtime for individual measurement tasks, TR 102 678 [i.8] describes the concept of Fixed
Data Transfer Time QoS (FDTT-QoS) parameters for data measurements, especially for FTP and HTTP data transfers.
NOTE: The "windowed" approach cannot be used to determine valid session failure ratios for the service used
since the data transfer is interrupted after a given time.
4.7.3 Bias effect for "best effort" and "windowed" approaches
With the "best effort" approach measurement locations/areas with higher achievable throughput contribute more
measurement samples than those with lower achievable throughput. "Slow" samples have just a slight effect on the
mean user data rate calculated from all measurement samples.
With the "windowed" approach measurement locations/areas with higher achievable throughput contribute in the same
way as those with lower achievable throughput.
4.7.4 First example
Figure 5 shows an example comparing "best effort" and "windowed" approach for one network.

Figure 5: Comparison of "best effort" and "windowed" approaches
ETSI
14 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
For the sake of simplicity assume a network providing only two dedicated speeds: 100 kbit/s and 2 500 kbit/s
(e.g. EDGE and HSPA). The graph shows the mean value of the user data rate calculated from (virtual) measurements
in this network with a certain percentage x of time in "slow" mode (100 kbit/s) and (1-x) in "fast" mode (2 500 kbit/s).
Observations:
• With the "best effort" approach the slow parts of the network affect the mean user data rate only in a
significant way if more than half of the time was spent in "slow" mode.
• With the "windowed" approach slow parts of the network affect the mean user data rate proportionally to the
time spent in "slow" mode.
4.7.5 Conclusions from first example
With the "best effort" approach the mean user data rate measured in networks providing "medium" and "high" speed
(e.g. pure 3G network) can be lower than the mean user data rate measured in networks providing "slow" and "high"
speed (e.g. 2G/3G network).
The reason is that even if both networks are in "high" speed mode for the same duration the higher number of "medium"
speed samples collected in the remaining measurement time in one network contribute more to the mean value than the
lower number of "slow" speed samples in the other network.
4.7.6 Second example
The second example compares "best effort" and "windowed" approaches for two networks, as shown in Figure 6.

Figure 6: Comparison of approaches for two networks with different speeds
For the sake of simplicity assume the same network (1) as before providing only two dedicated speeds: 100 kbit/s and
2 500 kbit/s (e.g. network with EDGE and HSPA). Assume a second network (2) also with only two speeds: 800 kbit/s
and 2 500 kbit/s (e.g. only HSPA).
The graph shows the mean value of the user data rate calculated from (virtual) measurements in these networks with a
certain percentage x of time in "slow" mode (100 kbit/s for network 1 and 800 kbit/s for network 2).
Observations:
• With the "best effort" approach, network 2 having a complete 3G HSPA coverage and showing the same
amount of time the same high speed is evaluated worse than network 1 having EDGE/HSPA coverage.
• With the "windowed" approach network 2 is evaluated better than network 1.
ETSI
15 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
4.7.7 Conclusions from second example
The "best effort" approach is common and best practice for assessing end-to-end Quality of Service parameters. A big
advantage is the fact that end-user-focused test cases can be created easily, e.g. download of a file or web page with
typical content. Thus the "best effort" approach is the best practice method for testing particular services
(web browsing, video sharing websites, email, etc.).
But one should act with caution when concluding from "best effort" measurements to user experience or just to
expected performance of other services than measured. The "windowed" approach should be used in addition to the
"best effort" approach in order to assess overall network performance and to conclude upon the expected
service-independent overall user experience.
4.7.8 Combining "best effort" and "windowed" approach
There is the possibility to combine the implementation ease of the "best effort" approach and the fairness of the
"windowed" approach by applying a "windowing" or "time binning" to the data collected from a best-effort
measurement. Both over- and under-sampling approaches are possible, as long as the final goal of assessing the
downloaded data amount within a certain time window is reached.
NOTE: Another way of combining the advantage of "fair calculation" by averaging transfer times and having
"meaningful data rates" and "service failure rates" is to perform a best effort approach, average the
transfer times achieved and recalculate the average throughput from the average transfer time in the sense
of an adjusted mean. Please also refer to clause 6.1.
4.8 Upper bounds for TCP throughput measurements
In many cases the maximum achievable throughput can be estimated. For instance, in case of TCP the maximum
throughput per socket is limited by the following equation:

This formula reflects the statement that there cannot be more unacknowledged traffic in the network for one socket than
the maximum TCP receive window size. For further details, please refer to e.g. RFC 793 [i.11], RFC 1323 [i.12] and
RFC 3481 [i.13].
A better estimation for the stationary TCP throughput per socket can be gained from delay, packet loss, and the
knowledge of congestion avoidance procedures. In the following equation, MSS stands for maximum segment size and
P stands for the probability of packet loss. For further details, please refer to the paper [i.14].
loss
5 Measurement environment aspects
5.1 Client/Server hardware
Any hardware being used to perform throughput measurements might have an impact on the overall results affecting the
achievable throughput. This holds especially true for the used measurement system, including any client and server side
hardware.
Depending on the used hard- and software components of the measurement system, throughput measurements can be
impacted by the amounts of measurement data that are written to a storage medium. Extreme throughput can cause high
CPU load as well as a high CPU load can have an impact on the maximum achievable throughput. Each hardware
component (CPU, hard disk, motherboard, etc.) can influence the total performance of a measurement system.
ETSI
16 Final draft ETSI EG 203 165 V1.1.1 (2012-02)
Thus the measurement system used should provide the means to send, collect and store all relevant measurement data,
including related logging data of the system itself, in a reliable way.
In case of passive measurements, storage medium and/or memory speed and/or size of the measurement equipment
limits the number of data that can be stored during data acquisition.
In case of active measurements the data upload and/or download is usually more IO-limited than CPU-limited.
E.g. when the throughput value gets close to the hard disk speed, the whole upload and/or download can be slowed
down because of the speed limitations of the hard disk.
In such a scenario, a possible solution could be to avoid writing any received file to a hard disk on the client or the
server side. Of course if the sender cannot send the data with enough speed, this has to be solved before any
measurement takes place. In very high speed networks caching the measurement files in memory (RAM disks) can
solve the problem.
NOTE: In order to overcome IO-limitations, another option is to decrease the amount of data that is written to the
hard disk on the fly or by totally switching off certain logging information if the measurement system
allows for such optimization. Especially in cases where Performance Enhancement Proxies (PEP, also
called accelerators) are involved and used together with a performance enhancement client being part of
the measurement system, TCP tracing can usually be switched off. The reason for this is that these
software components often implement protocol optimization and encryption techniques making the TCP
traces useless with respect to throughput evaluation based on the definitions given in TS 102 250-2 [i.2].
Another option in case that high throughput values are expected during measurement is to use several storage media in
order to distribute hard disk activity. In such a scenario one hard disk could be used to store the received data whereas
another hard disk could be used to store logging information of the local measurement system itself. Further
information with respect to requirements for Quality of Service measurement equipment can be found in
TS 102 250-4 [i.4] where the mi
...

ETSI Guide
Speech and multimedia Transmission Quality (STQ);
Throughput Measurement Guidelines

2 ETSI EG 203 165 V1.1.1 (2012-04)

Reference
DEG/STQ-00162m
Keywords
3G, GSM, network, QoS, service
ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE

Tel.: [phone]  Fax: [phone]

Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à la
Sous-Préfecture de Grasse (06) N° 7803/88

Important notice
Individual copies of the present document can be downloaded from:
http://www.etsi.org
The present document may be made available in more than one electronic version or in print. In any case of existing or
perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF).
In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive
within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available at
http://portal.etsi.org/tb/status/status.asp
If you find errors in the present document, please send your comment to one of the following services:
http://portal.etsi.org/chaircor/ETSI_support.asp
Copyright Notification
No part may be reproduced except as authorized by written permission.
The copyright and the foregoing restriction extend to reproduction in all media.

© European Telecommunications Standards Institute 2012.
All rights reserved.
TM TM TM
DECT , PLUGTESTS , UMTS and the ETSI logo are Trade Marks of ETSI registered for the benefit of its Members.
TM
3GPP and LTE™ are Trade Marks of ETSI registered for the benefit of its Members and
of the 3GPP Organizational Partners.
GSM® and the GSM logo are Trade Marks registered and owned by the GSM Association.
ETSI
3 ETSI EG 203 165 V1.1.1 (2012-04)
Contents
Intellectual Property Rights . 5
Foreword . 5
Introduction . 5
1 Scope . 6
2 References . 6
2.1 Normative references . 6
2.2 Informative references . 6
3 Definitions, symbols and abbreviations . 7
3.1 Definitions . 7
3.2 Symbols . 7
3.3 Abbreviations . 7
4 General throughput measurement aspects . 8
4.1 Purpose of the measurement . 8
4.2 Throughput equation . 8
4.3 Throughput on different layers . 9
4.4 Throughput at different reference points . 11
4.5 Active and passive measurement . 11
4.6 Load generation for active measurement of IP throughput . 11
4.7 "Best effort" versus "windowed" approach . 12
4.7.1 "Best effort" approach. 12
4.7.2 "Windowed" approach . 13
4.7.3 Bias effect for "best effort" and "windowed" approaches . 13
4.7.4 First example . 13
4.7.5 Conclusions from first example . 14
4.7.6 Second example . 14
4.7.7 Conclusions from second example . 15
4.7.8 Combining "best effort" and "windowed" approach . 15
4.8 Upper bounds for TCP throughput measurements . 16
5 Measurement environment aspects . 16
5.1 Client/Server hardware . 16
5.2 Operating System . 17
5.3 Performance Enhancement Proxies . 17
5.4 Shared medium . 17
5.5 All traffic versus filtered traffic . 18
5.5.1 User traffic . 18
5.5.2 Multithreaded applications . 18
5.5.3 Aggregate traffic . 19
6 Campaign planning and evaluation . 19
6.1 Mean user data rate versus mean transfer time measurements . 19
6.1.1 Example . 20
6.1.2 Conclusion . 22
6.2 Throughput calculation based on time or traffic . 22
6.2.1 Non-sampled averaging . 22
6.2.2 Sampled averaging . 22
6.3 Busy hours or peak-off . 22
6.4 Working days or weekends . 23
6.5 Locations . 23
6.5.1 Mobility aspects . 23
6.5.2 Area categories . 23
6.6 Calculating sample mean . 23
7 Throughput measurement checklists . 23
ETSI
4 ETSI EG 203 165 V1.1.1 (2012-04)
Annex A: Analysing IP traces under different aspects . 28
A.1 Layer-based analysis . 28
A.2 User-based analysis . 28
A.3 Volume-based analysis . 28
A.4 Time-based analysis . 29
A.5 Examples . 29
History . 30

ETSI
5 ETSI EG 203 165 V1.1.1 (2012-04)
Intellectual Property Rights
IPRs essential or potentially essential to the present document may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (http://ipr.etsi.org).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality
(STQ).
Introduction
The main purpose of the present document is to help the reader understand and differentiate between various throughput
definitions and calculation methods described in the TS 102 250 series and technical reports produced by STQ.
This guide describes the different aspects (e.g. protocol-specific, measurement-environmental, statistical) which should
be considered during planning, execution and evaluation of throughput measurements in order to avoid the major
problems which can occur.
TS 102 250-2 [i.2] standardizes throughput QoS parameters for popular IP based services used in mobile networks from
the user's point of view. Based on these definitions TS 102 250-7 [i.6] defines a model for network quality. Finally,
TR 102 678 [i.8] introduces a new method of throughput calculation based on fixed data transfer times.
ETSI
6 ETSI EG 203 165 V1.1.1 (2012-04)
1 Scope
The present document focuses on aspects of throughput measurements and their evaluation, providing different
approaches. It contains factors, guidelines and background information that should be considered during selection,
planning, execution and evaluation of throughput measurements.
2 References
References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the
referenced document (including any amendments) applies.
Referenced documents which are not found to be publicly available in the expected location might be found at
http://docbox.etsi.org/Reference.
NOTE: While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.
2.1 Normative references
The following referenced documents are necessary for the application of the present document.
Not applicable.
2.2 Informative references
The following referenced documents are not necessary for the application of the present document but they assist the
user with regard to a particular subject area.
[i.1] ETSI TS 102 250-1: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 1: Assessment of Quality of Service".
[i.2] ETSI TS 102 250-2: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their
computation".
[i.3] ETSI TS 102 250-3: "Speech and Multimedia Transmission and Quality (STQ); QoS aspects for
popular services in mobile networks; Part 3: Typical procedures for Quality of Service
measurement equipment".
[i.4] ETSI TS 102 250-4: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in mobile networks; Part 4: Requirements for Quality of Service measurement
equipment".
[i.5] ETSI TS 102 250-6: "Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects
for popular services in GSM and 3G networks; Part 6: Post processing and statistical methods".
[i.6] ETSI TS 102 250-7: "Speech and multimedia Transmission Quality (STQ); QoS aspects for
popular services in GSM and 3G networks; Part 7: Network based Quality of Service
measurements".
[i.7] ETSI TR 102 607: "Speech Processing, Transmission and Quality Aspects (STQ); TCP IP Stack
Parameter Settings for Microsoft Windows XP and Microsoft Windows Vista; Comparison and
Recommendations".
[i.8] ETSI TR 102 678: "Speech and multimedia Transmission Quality (STQ); QoS Parameter
Measurements based on fixed Data Transfer Times".
ETSI
7 ETSI EG 203 165 V1.1.1 (2012-04)
[i.9] ETSI TR 102 807: "Speech and multimedia Transmission Quality (STQ); Process description for
the transaction view model".
[i.10] ITU-T Recommendation X.290: "OSI conformance testing methodology and framework for
protocol Recommendations for ITU-T applications - General concepts".
[i.11] IETF RFC 793: "Transmission Control Protocol".
[i.12] IETF RFC 1323: "TCP Extensions for High Performance".
[i.13] IETF RFC 3481: "TCP over Second (2.5G) and Third (3G) Generation Wireless Networks".
[i.14] M. Mathis, J. Semske, J. Mahdavi, and T. Ott: "The macroscopic behavior of the TCP congestion
avoidance algorithm." Computer Communication Review, 27(3), July 1997.
3 Definitions, symbols and abbreviations
3.1 Definitions
For the purposes of the present document, the following terms and definitions apply:
application: software, using a particular service for providing related functionality to the user
Point of Control and Observation (PCO): point within a testing environment where the occurrence of test events is to
be controlled and observed
NOTE: A PCO is identified by a) a reference point or interface and b) a service access point (SAP) at the
specified reference point or interface, indicating unambiguously where (usually in a protocol stack)
events are observed (or other measurements are made). See ITU-T Recommendation X.290 [i.10].
service: capability of a specific layer and the layers beneath to provide a set of functions to higher layers
NOTE: One example of the higher layers is the application layer.
transmission capacity: maximum achievable throughput, which is determined by the physical characteristics of the
transmission media
NOTE: Examples of the physical characteristics of the transmission media are transmission capacity, applied
modulation and coding scheme.
3.2 Symbols
For the purposes of the present document, the following symbols apply:
Δt Predefined measurement time period
d
3.3 Abbreviations
For the purposes of the present document, the following abbreviations apply:
BMC Broadcast/Multicast Control
CDF Cumulative Distribution Function
CPU Central Processing Unit
FDTT Fixed Data Transfer Time
FTP File Transfer Protocol
FTTX Fiber To The X (of any type)
GGSN Gateway GPRS Support Node
HDD Hard Disk Drive
HSPA High Speed Packet Access
HTTP HyperText Transfer Protocol
ETSI
8 ETSI EG 203 165 V1.1.1 (2012-04)
IP Internet Protocol
ISO International Organisation for Standardization
MAC Medium Access Control
MSS Maximum Segment Size
NAT Network Address Translation
NP Network Performance
OSI Open System Interconnection
PCO Point of Control and Observation
PDCP Packet Data Convergence Protocol
PEP Performance Enhancement Proxy
PHY PHYsical layer
QoS Quality of Service
RAM Random Access Memory
RLC Radio Link Control
RRC Radio Resource Control
RTP Real Time Protocol
RTSP Real Time Streaming Protocol
RTT Round-Trip Time
SDP Session Description Protocol
SIM Subscriber Identity Module
SMTP Simple Mail Transfer Protocol
SYN TCP synchronise flag
TCP Transmission Control Protocol
TCP/IP Transmission Control Protocol / Internet Protocol
UDP User Datagram Protocol
UMTS Universal Mobile Telecommunications System
USB Universal Serial Bus
4 General throughput measurement aspects
When measuring throughput, certain protocol-specific aspects always have an impact on measurement results. Thus, the
purpose of the measurement and the respective measurement methodology has to be taken into account when evaluating
and comparing measurement results. The specific protocol used to perform a measurement may have an impact on
measurement results. In particular, throughput measurement results obtained using one protocol cannot generally be
assumed to be transferrable to other protocols. For example, performing application layer measurements in two different
mobile networks using both FTP and HTTP can result in a situation where one network produces better results for FTP
measurements while the other one produces better HTTP results.
4.1 Purpose of the measurement
The three main parameters to characterise the actual performance of an IP network are delay, packet loss and
transmission capacity.
The main cause why throughput is measured is to determine the achievable portion of the transmission capacity, being
influenced by delay and packet loss, for different services. From a user's point of view the throughput is a key
performance measure for a dedicated service perceived by the user on application level.
There is a notable difference between network throughput and application layer throughput which allows statements
about Network Performance (NP) and Quality of Service (QoS), respectively. A proper disambiguation of the different
perspectives can be found in clauses 5.1 to 5.3 of TS 102 250-1 [i.1].
4.2 Throughput equation
Throughput can be calculated using the following equation.
Amount Of Transferred Data
Throughput =
Duration
ETSI
9 ETSI EG 203 165 V1.1.1 (2012-04)
The achievable IP transmission capacity is a highly variable stochastic parameter that varies in both small and large
timescales. Every aspect of the throughput value has to be carefully listed; otherwise it can lead to misunderstandings.
4.3 Throughput on different layers
During packet transmission many protocols are involved. On every protocol stack layer the throughput metric can be
interpreted. While from a user's point of view only the highest protocol stack layer is important, from a network
operator's point of view the throughput on every protocol stack layer has its own meaning.
Depending on the used layer different PCOs apply and thus, the respective measurement results need to be treated
accordingly. The concept of PCOs is explained in clause 7.1 of TS 102 250-1 [i.1] and in clause 5.1 of TR 102 807 [i.9].
For network throughput measurements, different measurement environments and also different tools are needed to be
compare to application layer measurements, since different protocol layers are involved accessible at their own
reference points. Thus, the protocol layer has to be chosen and protocol-specific aspects like e.g. stationary state for
TCP have to be taken into account. For performance assessments of different services with respect to these network
characteristics, corresponding application traffic has to be generated and measured on a network with "live" overall
traffic load. Network throughput measurements and application layer throughput measurements cannot be mixed, even
though both use service protocols such as FTP or HTTP.

Figure 1: Idealistic view: ISO OSI Reference Model
The ISO OSI Reference Model is shown in Figure 1. The relation between the different protocol layer throughputs is
typically the following:
Lower Layer Throughput ≥ Higher Layer Throughput

NOTE 1: Due to headers on higher layers the lower layer throughput can normally be assumed to be greater than
higher layer throughput, i.e. ">" and not "≥".
Since every newly introduced layer increases the overhead, the amount of transferred data needed just for the
communication increases, thus decreasing protocol efficiency:
Usable Traffic
Protocol Efficiency =
All Traffic
ETSI
10 ETSI EG 203 165 V1.1.1 (2012-04)
NOTE 2: As stated above, higher layers include headers which add traffic and additional data to the transferred data
which is not directly available or usable for the user.
NOTE 3: In the real world there are exceptions from the guidelines stated above as described in Figure 2 (real
world example). Certain protocols, e.g. the Packet Data Convergence Protocol, are designed to increase
protocol efficiency as compared to higher layers by performing a header compression with the goal of
efficiently using scarce radio resources. Another example is the use of optimizer/compression software
that can modify user data in order to increase the perceived throughput. In this case one can also say that
lower layer throughput is bigger than higher layer throughput.

NOTE: Red circles indicate examples of header/payload compression.

Figure 2: Real world example: UMTS Protocol Stack (simplified)
with internet applications and possible optimization
Therefore, the throughput performance of an end-to-end communication is usually different from a user's point of view
compared to a network's point of view. In IP environments, upper layer protocols are executed in the end points
(e.g. terminals) while the network only performs a transport function involving only lower layer protocols. The end user
who is affected by the whole end-to-end path – including terminals - has to deal with the full protocol stack while the
network only deals with a subsection of the path involving only a few protocol layers.
Reporting just the measurement results without providing information about the layer involved and the protocol used
and the purpose of the measurement, e.g. network or application measurement, can lead to misinterpretation.
Furthermore, different service protocols use the underlying transport protocols differently and should thus be reported,
as well. For further details, please refer to clause 5.5.2.
ETSI
11 ETSI EG 203 165 V1.1.1 (2012-04)
In certain testing environments a service will not or cannot be evaluated by trigger points defined on just one layer. Start
and end trigger points can for example be on different layers or need to be mapped to other layers. For example, when
using a Performance Enhancement Proxy (PEP) the mapping of trigger points may refer to a point between the network
and the application layer, most commonly the transport layer. A detailed look at the influences of PEPs can be found in
clause 4.2.1 of TS 102 250-2 [i.2].
4.4 Throughput at different reference points
From a user's perspective the achievable throughput on the end-user side is important. This metric can be quite different
if it is applied at any other point of the network.
In case of TCP for instance, a connection terminates in two dedicated points from the user's point of view, e.g. the PC
running the web browser that is accessing a web server on a remote machine. From the network point of view, many
intermediate connections and therefore many more endpoints can be found, e.g. routers or firewalls. The different
properties of these TCP connections (traffic, retransmission rate) will result in different throughputs.
Performing network throughput measurements against these endpoints (destination hosts) can help finding bottlenecks
in the network or to compare different IP aggregation points along the service line. As an example a feasible
measurement scenario for testing different GGSN performance can be to measure the achievable TCP transmission
capacity against a FTP server located at the GGSN's Gi interface. Another measurement scenario would be to test the
performance of firewalls or NAT or L3-L7 deep packet inspection tools along the service line by placing an FTP server
before and after these equipments and comparing the measurement results. Measurement results for different destination
hosts on different logical places along the network should not be aggregated.
4.5 Active and passive measurement
An active measurement generates specific traffic on the network under test and is commonly used on access level. A
passive measurement captures "live" traffic at dedicated points in the network. This type of measurement is usually used
in the core network.
It is difficult to calculate the achievable throughput per user from the captured "live" traffic in the core network. For
such passive measurements it is e.g. not known if the user did not want to or could not generate more traffic on the
access side due to technical limitations of the client applications. Furthermore, the user behaviour is not predictable, but
has direct impact on the calculated throughput.
When using an active measurement to determine the core link capacity in terms of the maximum achievable throughput,
it should be considered that there usually is additional user and/or signalling traffic on the core link. In such a scenario,
the active measurement will measure the throughput that is achievable with the remaining (free) network capacities not
already allocated to other resources/traffic.
Commonly, active throughput measurements are performed on access network side generating traffic in order to
saturate the available transmission capacity. Passive throughput measurements, on the other hand, are more common in
the core network.
4.6 Load generation for active measurement of IP throughput
There are several ways to evaluate the IP network throughput from an end-user's perspective. Most commonly the IP
network throughput is measured based on UDP or TCP since it cannot be measured directly on layer 3. In IP networks
the respective traffic generation is usually done using TCP, because the majority of applications nowadays use this
protocol to transfer data. UDP can also be used for throughput measurements, since the measured UDP throughput
represents the upper bounds for TCP throughput in case the IP packets are in the same QoS class.
TCP was designed to adapt to the actual IP transmission capacity and to avoid congestion. To this end TCP provides
mechanisms such as e.g. slow start, fast retransmit and fast recovery. In addition TCP constantly monitors the actual
RTT and packet loss rates and uses those values to optimize the transmission capacity utilisation. For the setup and the
evaluation of the measurement many protocol specific aspects should be considered with respect to TCP, such as e.g.:
• TCP/IP stack parameters;
• stationary state of TCP;
ETSI
12 ETSI EG 203 165 V1.1.1 (2012-04)
• RTT;
• packet loss.
The aspects listed above influence the achievable throughput. For further information concerning the TCP/IP stack
parameters, please refer to TR 102 607 [i.7].
When speaking about throughput a stationary state is assumed, i.e. the TCP congestion control mechanisms had enough
time to adapt to the actual capacity of the network.
Usually the stationary state throughput itself reflects the throughput TCP can reach in the long run, with the time it takes
to reach it also being important. To assess how fast users can access network resources, measurements with small files
are recommended due to the fact that the TCP slow-start has a stronger effect compared to large files.
A common way to generate TCP traffic is by using FTP or HTTP. In case of the evaluation of the achievable
throughput the focus is not on the service evaluation, i.e. the service is used in order to generate traffic. However, it has
to be taken into account that other network specific aspects can influence the achievable throughput for different
services and thus the achievable IP throughput, such as e.g.:
• protocol-dependent routing;
• intermediate nodes, e.g.:
- PEP;
- NAT;
- firewalls.
When the traffic generator, e.g. some FTP server, is able to send out data packets with a higher rate than the network
can transmit the difference between the different application layer protocols is negligible assuming that any Layer 7
analyser running in the network analyses the packets at the same speed and TCP has achieved a stationary state. E.g. all
IP services using TCP as a transport service should provide nearly the same results when TCP has achieved a stationary
state. This might not necessarily be true when QoS classes are used on the Internet and in access networks (mobile,
xDSL, FTTX) assigning services to different service classes, for which specific service profiles might be defined. With
respect to this, the protocol used to measure the achievable IP throughput for a specific service class has to be chosen
carefully, e.g. FTP for background traffic, HTTP for interactive traffic and RTP for real-time traffic.
In case of UDP the measurement is more straightforward as UDP does not use any throughput control mechanisms like
TCP does: packets are sent by the sender without any regard of the capabilities of the intermediate network or the
receiver, i.e. the sender will not adapt to the actual transmission capacity and the measurement only counts the received
bytes.
From a test methodology's point of view the packet send rate should be chosen carefully and has to be higher than the
expected achievable transmission capacity of the network under test. Also, the expected packet loss should be
considered.
Although a UDP measurement provides a very good estimation for IP throughput if the measurement is well designed,
it does not provide any usable estimation for TCP throughput. The reason behind this is that no RTTs and packet loss
rates are measured and no traffic and congestion control functions will be taken into account. In addition network
elements such as traffic shaper or profiler usually only influence TCP traffic by dropping packets, but not UDP traffic.
4.7 "Best effort" versus "windowed" approach
In the following clauses the "best effort" and "windowed" approaches are introduced and compared using two
examples.
4.7.1 "Best effort" approach
The "best effort" approach is a best-practice method to achieve a maximum of measurement samples for a fixed test file
size within a given time. Measurement locations/areas with higher achievable throughput contribute more measurement
samples than those with lower achievable throughput. The mean value of all measured speed samples would be biased
by this effect.
ETSI
13 ETSI EG 203 165 V1.1.1 (2012-04)
Using the "best effort" approach as many data transfers as possible are measured within a given time. The number of
transfers depends on data transfer speed whereas the transferred amount of data per transfer is constant (not considering
retransmissions).
Figure 3: "Best effort" approach
Figure 3 illustrates the best effort approach. A number of subsequent transfers is measured but not all transfers last for
the same duration.
4.7.2 "Windowed" approach
Using the "windowed" approach, data transfers are executed within measurement windows of predetermined length.
The number of transfers is constant and the transferred amount of data per transfer depends on the data transmission
ratio.
Figure 4: "Windowed" approach
Figure 4 illustrates the windowed approach. Here it can be seen that all transfers last for the same duration, i.e. the
configured window size.
The "windowed" approach is best suited to achieve a more regular kind of sampling for network performance. As the
sampling is regular in the time domain one could approximately achieve a regular geographical sampling by doing tests,
e.g. as drive test with low but constant velocity. The mean value of all measured transfer speeds would better reflect the
average experienced speed under the assumption that users use the network at random locations.
As in reality network usage is not random or evenly distributed onto the area a geographic traffic distribution would
have to be considered to achieve a mean speed value even better reflecting user experience.
With respect to the findings above and especially in order to fulfil the common request for many QoS measurements to
have a limited and regular runtime for individual measurement tasks, TR 102 678 [i.8] describes the concept of Fixed
Data Transfer Time QoS (FDTT-QoS) parameters for data measurements, especially for FTP and HTTP data transfers.
NOTE: The "windowed" approach cannot be used to determine valid session failure ratios for the service used
since the data transfer is interrupted after a given time.
4.7.3 Bias effect for "best effort" and "windowed" approaches
With the "best effort" approach measurement locations/areas with higher achievable throughput contribute more
measurement samples than those with lower achievable throughput. "Slow" samples have just a slight effect on the
mean user data rate calculated from all measurement samples.
With the "windowed" approach measurement locations/areas with higher achievable throughput contribute in the same
way as those with lower achievable throughput.
4.7.4 First example
Figure 5 shows an example comparing "best effort" and "windowed" approach for one network.
ETSI
14 ETSI EG 203 165 V1.1.1 (2012-04)

Figure 5: Comparison of "best effort" and "windowed" approaches
For the sake of simplicity assume a network providing only two dedicated speeds: 100 kbit/s and 2 500 kbit/s
(e.g. EDGE and HSPA). The graph shows the mean value of the user data rate calculated from (virtual) measurements
in this network with a certain percentage x of time in "slow" mode (100 kbit/s) and (1-x) in "fast" mode (2 500 kbit/s).
Observations:
• With the "best effort" approach the slow parts of the network affect the mean user data rate only in a
significant way if more than half of the time was spent in "slow" mode.
• With the "windowed" approach slow parts of the network affect the mean user data rate proportionally to the
time spent in "slow" mode.
4.7.5 Conclusions from first example
With the "best effort" approach the mean user data rate measured in networks providing "medium" and "high" speed
(e.g. pure 3G network) can be lower than the mean user data rate measured in networks providing "slow" and "high"
speed (e.g. 2G/3G network).
The reason is that even if both networks are in "high" speed mode for the same duration the higher number of "medium"
speed samples collected in the remaining measurement time in one network contribute more to the mean value than the
lower number of "slow" speed samples in the other network.
4.7.6 Second example
The second example compares "best effort" and "windowed" approaches for two networks, as shown in Figure 6.
ETSI
15 ETSI EG 203 165 V1.1.1 (2012-04)

Figure 6: Comparison of approaches for two networks with different speeds
For the sake of simplicity assume the same network (1) as before providing only two dedicated speeds: 100 kbit/s and
2 500 kbit/s (e.g. network with EDGE and HSPA). Assume a second network (2) also with only two speeds: 800 kbit/s
and 2 500 kbit/s (e.g. only HSPA).
The graph shows the mean value of the user data rate calculated from (virtual) measurements in these networks with a
certain percentage x of time in "slow" mode (100 kbit/s for network 1 and 800 kbit/s for network 2).
Observations:
• With the "best effort" approach, network 2 having a complete 3G HSPA coverage and showing the same
amount of time the same high speed is evaluated worse than network 1 having EDGE/HSPA coverage.
• With the "windowed" approach network 2 is evaluated better than network 1.
4.7.7 Conclusions from second example
The "best effort" approach is common and best practice for assessing end-to-end Quality of Service parameters. A big
advantage is the fact that end-user-focused test cases can be created easily, e.g. download of a file or web page with
typical content. Thus the "best effort" approach is the best practice method for testing particular services
(web browsing, video sharing websites, email, etc.).
But one should act with caution when concluding from "best effort" measurements to user experience or just to
expected performance of other services than measured. The "windowed" approach should be used in addition to the
"best effort" approach in order to assess overall network performance and to conclude upon the expected
service-independent overall user experience.
4.7.8 Combining "best effort" and "windowed" approach
There is the possibility to combine the implementation ease of the "best effort" approach and the fairness of the
"windowed" approach by applying a "windowing" or "time binning" to the data collected from a best-effort
measurement. Both over- and under-sampling approaches are possible, as long as the final goal of assessing the
downloaded data amount within a certain time window is reached.
NOTE: Another way of combining the advantage of "fair calculation" by averaging transfer times and having
"meaningful data rates" and "service failure rates" is to perform a best effort approach, average the
transfer times achieved and recalculate the average throughput from the average transfer time in the sense
of an adjusted mean. Please also refer to clause 6.1.
ETSI
16 ETSI EG 203 165 V1.1.1 (2012-04)
4.8 Upper bounds for TCP throughput measurements
In many cases the maximum achievable throughput can be estimated. For instance, in case of TCP the maximum
throughput per socket is limited by the following equation:
TCP ReceiveWindow Size
Throughput ≤
RTT
This formula reflects the statement that there cannot be more unacknowledged traffic in the network for one socket than
the maximum TCP receive window size. For further details, please refer to e.g. RFC 793 [i.11], RFC 1323 [i.12] and
RFC 3481 [i.13].
A better estimation for the stationary TCP throughput per socket can be gained from delay, packet loss, and the
knowledge of congestion avoidance procedures. In the following equation, MSS stands for maximum segment size and
P stands for the probability of packet loss. For further details, please refer to the paper [i.14].
loss
MSS
Throughput 〈
RTT P
loss
5 Measurement environment aspects
5.1 Client/Server hardware
Any hardware being used to perform throughput measurements might have an impact on the overall results affecting the
achievable throughput. This holds especially true for the used measurement system, including any client and server side
hardware.
Depending on the used hard- and software components of the measurement system, throughput measurements can be
impacted by the amounts of measurement data that are written to a storage medium. Extreme throughput can cause high
CPU load as well as a high CPU load can have an impact on the maximum achievable throughput. Each hardware
component (CPU, hard disk, motherboard, etc.) can influence the total performance of a measurement system.
Thus the measurement system used should provide the means to send, collect and store all relevant measurement data,
including related logging data of the system itself, in a reliable way.
In case of passive measurements, storage medium and/or memory speed and/or size of the measurement equipment
limits the number of data that can be stored during data acquisition.
In case of active measurements the data upload and/or download is usually more IO-limited than CPU-limited.
E.g. when the throughput value gets close to the hard disk speed, the whole upload and/or download can be slowed
down because of the speed limitations of the hard disk.
In such a scenario, a possible solution could be to avoid writing any received file to a hard disk on the client or the
server side. Of course if the sender cannot send the data with enough speed, this has to be solved before any
measurement takes place. In very high speed networks caching the measurement files in memory (RAM disks) can
solve the problem.
NOTE: In order to overcome IO-limitations, another option is to decrease the amount of data that is written to the
hard disk on the fly or by totally switching off certain logging information if the measurement system
allows for such optimization. Especially in cases where Performance Enhancement Proxies (PEP, also
called accelerators) are involved and used together with a performance enhancement client being part of
the measurement system, TCP tracing can usually be switched off. The reason for this is that these
software components often implement protocol optimization and encryption techniques making the TCP
traces useless with respect to throughput evaluation based on the definitions given in TS 102 250-2 [i.2].
ETSI
17 ETSI EG 203 165 V1.1.1 (2012-04)
Another option in case that high throughput values are expected during measurement is to use several storage media in
order to distribute hard disk activity. In such a scenario one hard disk could be used
...