Railway traffic in Europe has increased over the past decade, and it is the aim of European policy makers to further promote the shift of passenger and cargo traffic from road to rail. However, to make railways the backbone of an efficient, sustainable and cohesive multimodal mobility system, the overall performance of the railway system has to improve. Even though everyone agrees on this, the very definition of performance and especially its measurement remains controversial.
At the 12th Florence Rail Forum the European Commission, representatives of railway undertakings, infrastructure managers, regulators, manufacturers, associations, as well as leading academics had the chance to discuss the different aspects of the performance of the European railway system. Furthermore, and thanks to the contribution of leading expert from the European Commission, we compared and confronted the peculiarities of the already existing scheme for Air Navigation Service Providers of the air transport sector with the possible evolution of the performance scheme of the railway sector.
Download now the summary of the presentations and the European Transport Regulation Observer. On the dedicated webpage, all presentations are available!
[accordion tag=h2]
[accordion-item title=”Highlights” id=1]
The European Rail Sector has room for improvements with regards to its performance. But how exactly are the railways performing and how are they improving? Key Performance Indicators are a popular tool to measure and benchmark the performance of an organisation. They are used on the company management level but increasingly also as a regulatory tool: most prominently the Single European Sky imposed a performance regime to improve the performance of the European Air Navigation Service Providers – the ‘infrastructure managers’ of the air.
Is this applicable to railways? Not so fast is the short answer. The 12th Florence Rail Forum discussed where we stand in terms of measuring and comparing the performance of the European railways. The controversial discussion in which the European Commission, regulators, operators and interest groups took part showed that everyone agrees on the need to improve performance but not necessarily on what performance actually means and how it could be measured.
[/accordion-item]
[accordion-item title=”Regulating the performance of European Railways: what can we learn from air transport?” id=2]
The 12th Florence Rail Forum took on a difficult and controversial topic, yet one which is already on the Commission’s agenda: the question of regulating the performance of European Railways. But, what is exactly railway performance? What exactly should be regulated and how should this be done concretely?
The idea that is guiding the European Commission is to proceed by analogy to air transport, where, in the context of the creation of a Single European Sky (SES), a performance scheme has been set up in 2004 (Regulation 549/2004) defining mandatory performance targets for the different European Air Navigation Services Providers (ANSPs). This analogy is certainly worth exploring and perhaps even translatable into the European railway sector. Yet, it is certainly also worth to do some bold thinking before rushing into exporting an approach that is arguably working in air to rail. I will do this in three steps: I will first discuss what, in my mind, rail performance should be about. I will then raise the question of the conditions for such performance. Finally, I will critically discuss the translation of the air approach into European railways.
What is rail performance?
Let us start from the times when railways were national, vertically integrated monopolies, a time when, by the way, performance was not really an issue because the railway system was seen as a public good. Yet, at that time measuring and “regulating” rail performance would have been relatively easy: one would have simply had to define politically desirable “performance indicators”, an approach which, probably, would have been inspired by new public management philosophy, treating the integrated railway company as a more or less autonomous public entity. And this is still the approach that underlies the Boston Consulting Group’s rail performance index, whereby the performance of the main vertically integrated railway companies is compared along a series of indicators.
There is nothing wrong with this approach, except that many railway companies have, since, been vertically separated (to different degrees) and access competition is starting to take hold. So, what is performance now? What are the relevant actors whose performance needs to be measured (and perhaps even regulated)? Everybody would probably agree that the performance of infrastructure managers (IMs), train operating companies (TOCs) is worthwhile measuring (and it is done, at least to a certain degree). One should, however, also consider the performance of stations in the case they are autonomous entities or the performance of railway regulators, for that matter, as it certainly plays a key role in the performance of the overall railway system. But even more problematic is the fact that the performances of the different actors that make up a national or even the European railway system, added together, will not automatically lead to a performing system. So, should railway performance be thought from the perspectives of the different relevant entities that make up the European railway system, or should it not rather be thought from the perspective of the system as a whole and then, perhaps, be broken down into the different relevant actors that do play a significant role in the overall system? In the first case, this is, in my mind, not really a matter of regulation, but rather a matter of corporate governance, whereby the owners – or the service contractors (generally the government) – should set performance targets and hold management accountable against them. In the second case, the question is whether it even makes sense to set system-wide performance indicators, as no single actor can be held accountable for (not) achieving them, given that system performance is by definition a collaborative effort. Either way rail performance should be seen in the general context of mobility, since the railway system is, after all, part of the intermodal transport system.
Conditions for rail performance
This leads me to consider whether we should not rather pay attention to the conditions of the performance of the overall (national or even European) railway system, rather than to the performance of the system itself. At least when it comes to regulation.
By conditions for performance I mean, technical aspects such as interconnection and interoperability in the case of the railway sector and standardization as the main underlying condition. It is standards that will ultimately lead to the smooth integration of the European railways by way of harmonizing infrastructures, signaling, rolling stock, data exchange and many other things more. I would claim that the progress in matters of performance of the overall railway system, be it at a regional, national or at the European level, will be almost totally correlated with the progress made along these different standardization dimensions. So, should we not rather measure or even regulate the performance of standard developers on the one hand and monitor or even regulate the implementation of these standards on the other hand?
In the first case (standard development), we might again be inspired by the air transport sector namely the promotion of standardization research by way of SESAR (Single European Sky ATM Research) and the designation of a so-called “deployment manager” to make such research operational. In other words, this not so much a matter of regulation, but rather a matter of inspiring, facilitating, financing and then implementing standardization together with all involved stakeholders.
In the second case (standard implementation) I do indeed see a case for regulation, namely when it comes to mandating and ultimately regulating the adoption and implementation of technical standards by the different concerned operators, in particular IMs and TOCs, but probably even suppliers. This is typically the task of the European Rail Agency (ERA), which, perhaps, needs to be strengthened so as to perform this task.
Air and rail performance: analogy or more?
This leads me to the original thought of the European Commission, namely the ideas (1) to measure and (2) to regulate the performance of railways as it is currently done in air. Without going into details as to how this is done in the European air transport sector, it is nevertheless essential to recall the exact scope and subsequent limitations of such an approach: what we are talking about here are, as stated in the beginning, the ANSPs, which are the equivalent of the IMs in the railway sector. There is, indeed, much to be said about independently regulating the IMs along predefined key performance indicators (KPIs). However, these KPIs cannot simply be imported from air transport, as the nature of the rail network – namely its capillarity, along with the distinction of different types of networks (e.g., high-speed, long-distance, agglomeration) – is better compared with electricity or gas rather than air. Also when it comes to the method of defining, measuring and ultimately sanctioning performance, the approach – and the governance of the approach – chosen in the case of air offers significant room for improvement and would probably have to be redesigned in the case of rail. It is, among other things essential to “keep it simple”: as a first step, let us define and measure KPIs; then, if they work, more complex KPIs can be introduced.
However, the ultimate goal is entirely identical with the air transport sector, namely to achieve a performing, interoperable und ultimately fully integrated European rail infrastructure.
Matthias Finger | FSR-Transport Director
[/accordion-item][accordion-item title=”Key Performance Indicators – the need for formal statistical methods” id=3]
Chris Nash
Institute for Transport Studies
University of Leeds
Despite the reforms of the last 15 years, the rail industry remains heavily concentrated; all European countries still have a single public sector infrastructure manager in charge of most rail infrastructure, and in all except Britain a state owned passenger operator remains the largest (and in many countries the only) main line passenger operator. Whilst competition is more widespread in freight, one operator continues to have a large market share in most countries.
This lack of competition means that benchmarking to check cost efficiency and quality of service is of particular importance in the rail industry, and this is recognised in European legislation by the requirement that pressure should be exerted on the infrastructure manager to reduce costs and charges. Consequently, there is a strong interest in key performance indicators for the rail sector, both to track how performance changes over time and to benchmark it against best practice on a European scale.
Regarding cost efficiency, the obvious measure to use is cost per unit of output. But we immediately hit on a problem. Railways are multi output organisations, producing a range of passenger and freight services and it is not easy to identify a single measure of output. For instance, the cost per tonne kilometre of container traffic is likely to be higher than for coal, and the cost per passenger km of regional passenger traffic higher than that for inter city, simply because mean train loads are normally lower. So simple comparisons between countries with different balances of traffic may be misleading, and even comparisons for a single country over time may be dominated by changes in traffic mix as some markets grow and others decline.
Cost differences may also be explained by other factors, such as differing wage rates or geography; it is well established that both infrastructure and train operating costs will tend to be lower the higher the traffic density. So it is necessary to take account of such differences either through qualitative judgment or the use of statistical methods. In the latter approach, the key performance indicator becomes the difference between the cost level of the company in question and what the model predicts should be possible given its mix of traffic and the other circumstances it is in. Several techniques are available but it has been argued that the cost frontier approach, as for instance used in work for the British rail regulator ( Smith, A.S.J., Wheat, P.E. and Smith, G. (2010), ‘The role of international benchmarking in developing rail infrastructure efficiency estimates’, Utilities Policy, vol. 18, 86-93) has advantages.
Similar considerations apply to measures of quality of service. It is not surprising for instance that short distance operators tend to achieve higher punctuality than long. Several speakers at the 12 Florence Rail Forum referred to market share as the ultimate measure of performance. But market shares depend on a host of factors outside the control of rail managers, including traffic density, lengths of haul, historic levels of investment in road and rail infrastructure and charges for the use of roads and rail lines. Again the only answer seems to be the estimation of statistical models to estimate the impact of such factors on performance, so that a measure may be made of how far market share differs from what might be possible given the value of all these characteristics. Work of this nature is reported for instance in van de Velde, D., C. Nash, A. Smith, F. Mizutani, S. Uranishi, M. Lijesen and F. Zschoche (2012),”EVES-Rail – Economic effects of Vertical Separation in the railway sector”, Report for CER – Community of European Railway and Infrastructure Companies, Brussels.
Whatever approach is taken, it is obviously necessary to ensure reasonably consistent data for comparisons to be meaningful. The International Railways Statistics dataset collected by the UIC has been much used in academic studies. Unfortunately, the data only refers to UIC members, so new entrants are missing, and for those UIC members operating in several countries often refers to all their operations rather than just their operation in the home country. There are other sources of inconsistency as well (see van de Velde et al, op cit). Thus it is necessary to turn to other sources of data, including Eurostat, the rail market monitoring studies and annual reports and accounts. Again the use of statistical methods allows account to be taken both of random noise in the data and of specific biases of which the researcher is aware.
Without the use of statistical methods to allow for all these issues, the use of key performance indicators may be seriously misleading.[/accordion-item]
[accordion-item title=”How to define, measure, and improve the performance of the European railway system?” id=4] A summary of discussions
The topic of performance of the railways has been discussed for the first time at the Florence Rail Forum and, overall, the discussion showed that a performance based approach to regulation is still at a very early stage. The discussion addressed three questions:
- Which performance indicators are of relevance for each stakeholder and how are they measured?
- What are the core aspects of rail business where performance needs to be improved? How to create the right incentives?
- How should national and EU rail market legislation evolve to safeguard better performance?
- What can we learn from the experience of benchmarking the performance in the air transport sector?
Which performance indicators are of relevance for each stakeholder and how are they measured?
Everybody agrees that the performance of the railway system needs to be improved. Performance measurement and Key Performance Indicators (KPIs) has become a popular tool – not only in management but also as a policy instrument. Whereas in air transport a “performance scheme” is in place for the European Union, the rail sector does not have that (yet).
Of course KPIs are already widely used also in the railway sector on the company level. However, KPIs that can be used to benchmark the European railway system would have to be of a different nature than the ones used at a company level. In particular, this is due to the fact that different actors have different purposes for measuring performance:
- On the company level KPIs are mainly concerned with improving operational and financial performance of a railway undertaking.
- From the customer perspective performance is about something else; mainly price, availability and reliability of service.
- Transport authorities are mostly interest in the “value for money” under public service contracts: the service quality, the state of the network the cost for the tax payer.
- Regulators would measure accessibility and safety performance of the national railway system.
- EU institutions are concerned with interoperability, market access the optimal use of EU funds and the contribution of the railway sector to larger policy goals.
While KPIs as a regulatory tool are certainly a new topic, to some extent performance has always played a role in the railway regulation before. The Rail Forum started with some considerations on the state of play of the performance approach in railway regulation: while there are no performance targets on a European level for the railway system as a whole, performance is part of European railway regulation in several areas in technical regulation as well as market regulation. Examples for this include:
- Directive 2012/34/EU (Single European Railway Area) mentionsPerformance schemes to encourage RUs and IMs to minimise disruptions and improve the reliability and punctuality of services (Article 35, Annex VI)
- According to the 4th Railway Package IMs shall cooperate to monitor and benchmark performance and to contribute to rail market monitoring
- The TEN-T Guidelines (Regulation (EU) No 1315/213) define Core Network Corridors and foresee the monitoring of the relevant network development and activity in the multimodal corridors
One of the most important lessons of the discussion may have been that the development of KPIs takes a lot of time and has to follow several steps. Before KPIs can be identified there has to be a thorough process of monitoring first. The PRIME Group (Platform for Rail Infrastructure Managers in Europe) has worked for over two years to identify some indicators. This had followed a process from first defining the core topics for performance to then defining indicators to measure them. Clearly, indicators shall be useful and simple, yet the experience shows that this is easier said than done. The PRIME Group managed to narrow done a collection of over 100 indicators suggested by the industry to about 60. Yet both usefulness (is an indicator that works for long distance also relevant in the metropolitan area?) and simplicity (punctuality is of key importance, yet in Europe not even two countries measure it the same way) are hard to achieve.
It was however underlined that the long time the process in the framework of the PRIME Group was essential to building up the necessary trust between the actors. Trust is a crucial category as performance measurement relates to sensitive company data.
Data
One of the issues related to data is their availability. As a comparison the American example was mentioned during the discussion: the US Surface Transportation Board has been collecting operational data from the railways since 1969 on each single train travel. With the help of technologies that allow the analysis of big data this is now an important resource for modelling that is currently absent in Europe.
In Europe the situation is of course more difficult owing to the different national systems in place. It was pointed out that railway regulators should be the institutions collecting data yet the legal basis on which they can do that differs widely across Europe. For instance, it was pointed out that the right of regulators to retrieve data from railway companies is almost unrestricted in Poland whereas in Belgium hardly any such data is made available to the regulator.
An important question is of course what is measured and how. The current patchwork of approaches and measuring leads to an inconsistency also in terms of methods.
Better Analysis of Data
From an academic point of view the importance of the correct application of formal statistical and macro-economical tools was stressed. The performance of the complex railway system can only be measured correctly if these analytical tools are used to determine the influence of different variables on the performance. The variables that influence the different areas of performance are to a large extend far beyond the control of a railway operator or infrastructure manager and depend on such elements as geographical factors, political decisions or past investments. For instance the performance in terms of cost per kilometre depends on many factors such as the mix between passenger and freight transport, between high-speed, regional and metropolitan lines and wage levels. Statistical models can be used to determine the impact of each one of these variable in order to make an assessment about how good or bad a railway system is performing – under its given circumstances.
Who does what?
It was acknowledged that differernt actors work on railfoway performance and the efforts are ot coordinated. Yet at the current stage there is no plan for the commission to take the lead but rather to understand well the different activities and the areas they are looking at.
What are the core aspects of rail business where performance needs to be improved? How to create the right incentives?
Naturally the discussion on the question of where performance needs to be improved included a lot of (sometimes well known) arguments concerning current issues of the European railway system. In short it was consensus that the European railways have issues with cost, quality and market share. There are many areas that can be looked at in order to measure progress. From the operator side it is important to differentiate according to the context. KPIs need to be defined differently for:
- Monopoly situations: KPIs need to measure the performance of the infrastructure managers. Contractual arrangements defining performance targets between operators and infrastructure managers are a useful tool.
- Public service contracts: performance targets are already an important part of public service contracts and could be further harmonized as regards the areas they address.
- Open market competition: given the crucial role of the customer KPIs could actually be used to inform the end customer better. Such public KPIs would for instance illustrate the energy efficiency. They would have to be developed by the sector and monitored by a public authority.
Specifically the areas mentioned where performance can be improved and possibly measured are: In the area of reliability the measurement of punctuality was looked at more closely. Different standards exist across Europe but an interesting example came from Japan. There, a method is applied that allows a weighing of the effects of delays on passengers across the network as a performance category.
What can we learn from the experience of benchmarking the performance in the air transport sector and how should national and EU rail market legislation evolve to safeguard better performance?
The discussion has shown that a decision is needed on a set of simple measurable indicators that can be applied throughout the system. Such an exercise has already been accomplished in the air transportation sector that served as a case study during the Forum. European Air Navigation Service Providers (ANSPs) are subject to a performance regime that defines mandatory targets over a four-year reference period. The ANSPs can be compared to the infrastructure managers in railways: their performance shortens flight times and allows more direct routes thus improving fuel efficiency and the environmental impact. Cost reductions for ANSPs allow lower route charges for the airlines and ultimately reduce costs for passengers. The discussion made it however quite clear that it will not be possible to apply the same approach to the rail sector and from the side of the Commission there is no ambition to impose a comparably strict regime to railways. Yet one important lesson was learnt from air where the performance discussion started already around 1997. The number one challenge is the complexity. Therefore KPIs need to be simple in the first place. Once the easiest and most straightforward indicators are identified and agreed upon they should be measured across Europe. A more sophisticated regime can possibly follow after that.
None of the operators currently seem truly “at ease” when talking about KPIs. In fact, many of the comments had been rather inward looking and addressing most of all existing problems and ambiguities. There is however agreement on the necessity to align the different activities that are currently under way in different groups and different undertakings.
Even though operators are cautious in endorsing performance regulation there seems to be an overall positive attitude towards the idea of a more light handed approach to regulation: formulating performance targets can replace detailed and prescriptive regulation offering operators more flexibility.
[/accordion-item]
[accordion-item title=”Further reading” id=5]
Florence School of Regulation Transport Area, 2016, 12th Florence Rail Forum Summary of presentations
This document offers summaries of the presentations given by the participants of the 12th Florence Rail Forum “How to define, measure, and improve the performance of the European railway system?”.
The Forum extensively discussed the topic of performance and measurement, with representatives of the European institutions, local authorities, operators, manufacturers and regulators. Many of them had already tackled the issue, yet a systemic view of performance of the railway sector seemed to be lacking. Hence, this Forum favoured the cross-modal confrontation with the air sector, where the topic of performance and measurement of key performance indicators are already in use as part of a performance regime. The peculiarities of the rail sector have emerged as well as interesting lessons from aviation.
Florence School of Regulation Transport Area, 2014, “High-speed rail vs. low-cost air : competing or complementary modes?”, European Transport Regulation Observer, n. 2014/01
This issue of the European Transport Regulation Observer reflects upon the topic discussed at the 2nd Florence Intermodal Forum, where policy makers, regulators, associations and operators of both the rail and the air transport sector came together to discuss current issues that impact both sectors alike.
For the first time at our Forums discussion was juxtaposing two modes of transport, and the central question was whether high-speed rail and low-cost air were competing or complementing modes of transport. After the liberalization of the European air market low-cost air carriers had taken over many medium length routes in Europe. High-speed rail could win back a lot of ground on many routes and remains dominant especially on city connection as illustrated by the Rome-Milan corridor example. The development is taking place against the background of an ongoing liberalization process in the rail market which has an important effect on innovation in the sector. Are the two modes serving different markets or are they in competition with each other? To what extent can they be complementary? What role does state aid play for each sector and how can policy work to address passenger needs?
Masayuki, Matsumoto, 2013, “Changing RAMS for Railways: Proposals from Japan”, JR EAST Technical Review-No.25
This article provides an overview of the RAMS (reliability, availability, maintainability and safety) standard and possible revisions of the current standard, from the point of view of one of the most important Japanese railways companies.
In Japan, compliance to international standards such as International Electrotechnical Commission (IEC) standards is demanded with the Agreement on Technical Barriers to Trade (TBT) and Agreement on Government Procurement coming into force of at the World Trade Organization (WTO), which commenced in January 1995. Moves are currently underway to make European standards such as those of the European Committee for Electrotechnical Standardization (CENELEC) into international standards. And a typical example is the RAMS standard (IEC 62278). The RAMS standard is expected to greatly affect Japanese railways into the future. It applies specifically to railways, and it covers the entire lifecycle from system concept through to disposal. Japanese railways are renowned worldwide for their high levels of safety and stability. But if traditional methods of Japanese railway operation will differ from those of the RAMS standard, also Japanese railways will be forced to make corrections. This article thus covers an overview of the RAMS standard as well as current trends and Japan’s efforts.
Smith, A.S.J., Wheat, P.E. and Smith, G.,2010, ‘The role of international benchmarking in developing rail infrastructure efficiency estimates’, Utilities Policy, vol. 18, 86-93
International cost efficiency benchmarking played a central role in informing the Office of Rail Regulation’s (ORR) determination of Network Rail’s future funding during the 2008 periodic review (PR08) of the company’s finances. This paper sets out how international benchmarking can inform a regulator’s decisions on efficiency and, in particular, how international econometric studies can be used alongside other evidence in the regulatory context. We start by reviewing the use of previous international benchmarking work. We then set out the data, methodology and results in respect of the two separate econometric studies carried out as part of PR08. The further work that was done in support of the econometric results is then described. The paper shows that top-down econometric techniques, combined with bottom-up engineering analysis produced comparison between Network Rail and its peers. We conclude by outlining how the econometric results were used, in conjunction with other evidence, to reach a final efficiency determination, and how we consider that international benchmarking can be applied by other regulators.
The Boston Consulting Group, 2015, The 2015 European Railway Performance Index, The link between performance and public cost
In the first Railway Performance Index report, published in 2012, the BCG sought to understand what drives railway performance in general, with particular emphasis on whether there might be a link between high performance and market liberalization. (See The 2012 European Railway Performance Index: Understanding What Drives High Performance, BCG Focus, November 2012.) the BCG found only weak correlations between performance and the degree of liberalization and between performance and the choice of governance model. Their key finding was that a railway system’s overall performance typically correlates with the level of public cost, which we defined as the sum of public subsidies and investments in the system.
Their review in 2015 confirms the correlation between performance and public cost. To probe more deeply into this correlation, the BCG also examined the link between performance and how governments allocate public subsidies between infrastructure managers and train-operating companies. Significantly, they found a correlation between the value that countries derive from public cost and the percentage of public subsidies that are allocated to infrastructure managers. The key takeaway: countries that get the most value from public spending on railway systems also allocate the highest percentage of subsidies to infrastructure managers.
Smith, A.S.J., Wheat, P.E. and Smith, G. (2010), ‘The role of international benchmarking in developing rail infrastructure efficiency estimates’, Utilities Policy, vol. 18, 86-93
This paper sets out how international benchmarking can inform a regulator’s decisions on efficiency and, in particular, how international econometric studies can be used alongside other evidence in the regulatory context. It starts by reviewing the use of previous international benchmarking work. The paper shows that top-down econometric techniques, combined with bottom-up engineering analysis produced comparison between the British Regulator Network Rail and its peers. It concludes by outlining how the econometric results were used, in conjunction with other evidence, to reach a final efficiency determination, and how the authors consider that international benchmarking can be applied by other regulators.
[/accordion-item]
[/accordion]
