top of page

The future of intelligence is artificial-B-AIM PICK SELECTS

Artificial Intelligence has been around for decades, but is now reaching a stage of maturity where it could soon become a common tool used throughout the rail industry. David Burroughs looks at how AI is already in use and what widespread adoption could mean for the industry in the future.

ARTIFICIAL Intelligence, aka AI, is a phrase that’s been around since the 1950s, with roots that go back further. But the term can still be somewhat of a misnomer, a buzzword thrown around and slapped on any technology that processes and analyses some form of data to solve a problem or provide an output.

In amongst the fluff, solid work is underway around the world into deep and machine learning, taking computing to the next level, and looking at ways these specialised forms of AI can tackle problems in industries ranging from health care and banking to insurance and, importantly, transport.

AI is already capable of doing a lot of tasks: beating humans at logic-based games such as chess, Go, and Jeopardy!, along with more practical applications such as examining MRI scans to detect cancer at a faster and more reliable rate than humans can. AI is also guiding self-driving cars.

Dame Wendy Hall, regius professor of computer science at the University of Southampton in Britain, says AI has gone through several stages of development since its inception, with each new stage usually taking around 30 years to become entrenched in industry.

“There’s been a series of waves of AI,” Hall says. “From the early decision-making systems using trees to the expert decision makers and knowledge-based systems, then the development of the neural network in the 1980s, which has led to today’s work on deep learning and machine learning. After a wave of ideas and early development that doesn’t pay off, you get the AI winters where nobody funds AI research because it doesn’t deliver (immediately).”

“At the moment the AI systems are trained to do a particular thing in a particular area, so we’re miles away from general AI or AGI, which is what our brains are,”

Dame Wendy Hall, regius professor of computer science at the University of Southampton in Britain

AI can often conjure up images of powerful computers capable of performing a multitude of tasks better than humans. However, despite large strides made in the development of AI, we don’t need to begin worrying about Skynet-like computer systems taking over the world just yet.

“At the moment the AI systems are trained to do a particular thing in a particular area, so we’re miles away from general AI or AGI, which is what our brains are,” Hall says. “We’re far away from a machine that can think in a way we do and cross different subjects and make connections across different subject areas, let alone whether it could have a conscience or a soul. But we can begin to see how we might build those and that’s the next push of AI.”

The term AI is commonly used to refer to a range of technologies such as software, algorithms, processes, and robots that are able to acquire analytical capabilities and perform tasks as opposed to legacy machines that could only act on human inputs, shifting technology’s role from enabler to advisor.

Rail industry’s focus

The rail industry’s focus on AI has increased at a rapid pace in the past decade, so much so that when the European Union’s Shift2Rail (S2R) research project was first launched in 2014, AI – or digitalisation for that matter – wasn’t even mentioned in the original programme.

Both concepts are now very much ingrained in the project, says S2R head of research and innovation, Mr Giorgio Travaini, with digitalisation a necessary precursor to the introduction of AI. Once systems such as signalling are digitalised, they can then move towards being automated and integrated with other systems such as the traffic management system.

AI is now being utilised in a number of S2R projects. S2R has published a call specifically focusing on AI in its 2019 call for proposals, which Travaini says was deliberately left open to cover a wide range of potential topics applicable to the rail industry.

This includes using machine learning for obstacle detection and monitoring the state of infrastructure; mining big data; and using AI to detect cybersecurity intrusions, a topic becoming more and more relevant in the rail industry as the Internet of Things (IoT) becomes more ingrained in systems and processes.

“All these are not new, they are being used in other industries so we also want to leverage from what other sectors have done,” Travaini says.

As part of these projects, the research will study a number of potential impacts of the technology on the industry, including:

  • increased capacity

  • reduced life cycle costs

  • reduced errors from both humans and existing computer systems

  • improved efficiency and increased performance

  • high-level automation and auto-adaptive systems

  • simplified supervision and fast problem resolution

  • reduced complexity with simplified and interoperable interfaces, and

  • improved flexibility.

Much of the current focus on AI by the rail industry is based on the development of video-based analytics systems. These programs are able to take and process a video or photos in real time to identify a wide range of objects from tracks and rails to signals, pedestrians, cars, other trains, or even faults along the tracks.

The company has focused on developing systems capable of analysing and combining this data to provide services such as train positioning, which feed into the ATP.

Derel Wust , 4Tel managing director

4Tel, a software and hardware engineering firm based in Newcastle, Australia, has been exploring the use of AI to develop automatic train protection (ATP) systems for the New South Wales Country Regional Network (CRN), which is currently operated and maintained by John Holland on behalf of the NSW state government.

Rolling stock can now be equipped with a range of sensors, including GPS, inertial navigation, odometers, radar, Lidar, cameras, ultrasonic and acoustic monitors, generating a massive amount of data. 4Tel managing director, Mr Derel Wust, says the company has focused on developing systems capable of analysing and combining this data to provide services such as train positioning, which feed into the ATP.

The company, which has been working alongside the University of Newcastle Robotics Laboratory, launched a project titled “Horus” in reference to the all-seeing eye in Egyptian mythology. Wust says the project has developed its own systems, calculations and algorithms for rail-specific uses from scratch as those developed for other industries cannot necessarily be copy-and-pasted into rail-specific applications.

“The idea was that we could focus on the different elements, for example determining where the train is,” Wust says. “GPS works well but it’s not so good with multiple tracks, when you need to identify which precise track you are on.”

Video cameras

Using video cameras which provide 25 images a second, 4Tel is able to identify different tracks and objects alongside the track, which – when the position of these objects is known – allows it to triangulate the position of the train in real-time.

“An observation we came to quite quickly was that a lot of the infrastructure doesn’t actually change, so a bridge, a poll, or a sign were there yesterday and the day before,” Wust says. “If you’re able to identify where you are and that those things exist, you’ve got references to where you are using triangulation and measurement.”

4Tel looked at various rules-based systems, including those designed to read road vehicle number plates, but didn’t find one suitable for rail applications.

“The trouble with rules-based systems is they work but they don’t scale,” Wust says. “So when you’re adding the complexity of a sign or something you see on a railway, and you add in the complexity of day, night, fog, rain, light conditions and smoke, as far as the computer is concerned the images all look different. What we quickly found was that a rules-based system wasn’t quick enough or reliable enough, there were too many exceptions to the rules.”

“It’s a statistical process that keeps learning and the more signs it sees the better it gets,”

Derel Wust

The company instead turned to neural networks and machine learning, where the computer is shown dozens and dozens of images of the same sign, with each image differing slightly in parameters such as angle, time of day and lighting conditions. Following this process, which Wust says takes at least 60 images per object, the computer is then able to identify the sign with around 96.7% accuracy – a figure that continues to improve over time as more images are fed in.

“It’s a statistical process that keeps learning and the more signs it sees the better it gets,” Wust says. “Particularly in the rail industry where you run up and down the same track and you’re seeing the same infrastructure through different light conditions, your computer learning is actually quite good.”

4Tel is now applying this software to other applications and is able to identify non-static objects along the line such as pedestrians, cars and trains, providing information that is fed back into the ATP to advise and support the driver.

Other initiatives around the world to develop rail-applicable AI are also looking to video analytics as a means to provide a range of different services.

Research and development

Siemens’ 120-member research and development team, based in Berlin, has 30 people dedicated to the development of AI systems such as computer vision, machine learning, deep learning and data analytics, who work alongside teams with expertise in mechatronics, sensors and software engineering.

The biggest initiative currently underway is the Assisted and Autonomous Driving for Rail (AAR) project to develop assisted and autonomous rail vehicles. The project has developed a driver assistance system that can now be installed on LRVs, and also contributed largely to trials of an autonomous tram in the German city of Potsdam, which in May was expanded from the original 6km section of line in the city’s suburbs to 13km.

A converted Potsdam Transport (ViP) Siemens Combino LRV is operating at Grade of Automation 3 (GoA 3) with a driver present who is able to intervene if needed. The vehicles has so far operated without incident since May 2018.

Along with the train control, Siemens is also using the AI-powered video analytics software to develop intelligent CCTV, or iCCTV.

“With our AI we are adding analytics to the pure storing of the video,” explains Mr Claus Bahlmann, Siemens’ head of department, intelligent software, artificial intelligence and computer vision.

This includes calculating if a seat is occupied by a passenger or just a bag, or if a wheelchair space is occupied.

“If we know that an area is already taken by a wheelchair, this information can be sent to the platform and announced so another wheelchair user can be notified that the space they are waiting for is already occupied and they can take the time while waiting to move to the next available area,” Bahlmann explains.

The technology is also capable of detecting aggression. It can alert operators if a fight has broken out or if the train is being vandalised, allowing them to make an announcement in the affected vehicle or alert authorities.


AI has direct benefits for maintenance technicians, and could be one area where the technology might be more likely to become an important assistant for staff rather than a replacement.

“There is a breadth of different sensors that are mounted on the trains and the infrastructure such as point machines,” Bahlmann says. “At Siemens, we are capturing and using the sensory data – whether it is images or engine temperatures or measurements of the current in the point machines – and developing analytics solutions that monitor the state of these assets to predict what action is needed.”

Ten years ago, an employee would have taken a 1000-page manual with them when servicing a train, flicking through to find the information required for any given task. That changed with the introduction of tablets, which allowed technicians to instead scroll through a PDF of the manual and search for particular sections.

In the rail industry where you run up and down the same track and you’re seeing the same infrastructure through different light conditions, your computer learning is actually quite good.

Derel Wust

But that too could soon change. Siemens is developing applications that enable rolling stock engineers to use the camera on the tablet to view different parts of the train such as the cab and tap on different items such as a button, lever or screen. The tablet is then able to identify what the item is and provide any required information such as the correct page in the manual or a 3D CAD model of the part. It can also provide a one-click part ordering service.

“This obviously needs computer vision technology. We analyse the camera images to recognise which parts the tablet is pointing at and the user is selecting,” Bahlmann says.

The industry has spent years collecting data to develop predictive maintenance capabilities, so much so that it is now considering how to process it all and extract meaningful insights. AI could prove a solution by automating data acquisition, processing and analysis, producing meaningful outputs that are both useful and actionable.

One topic S2R is focusing on is creating data models where the data coming from one system can be translated so it is compatible with information from elsewhere.

“These subsystems don’t speak the same language, they don’t have the same model so we are researching a kind of supermodel on top of it which will not affect the model you are working on, but it will allow the different subsystems to communicate with each other,” Travaini says.


Safety is in the DNA of the rail industry, and before a new system can be introduced, it must comply with rigorous standards.

“The big problem with deep learning as a way to training your machines is that it’s very hard for the system to explain how it got an answer,” Hall says.

“With the traditional expert systems that were very rule-based, you could get the system to say what rules it used to get the answer. With deep learning it’s a lot more complicated than that because you are dealing with nested layers of networks over data, and you don’t know what route it has taken through the data.”

This is echoed by Thales chief technical officer for ground transportation activities, Mr Amaury Jourdan, who says that there is still some way to go before AI can be trusted to take control of a safety critical system.

“AI and safety are not compatible without significant adaptation,” he says. “Which doesn’t mean that it can’t be used at all but at this stage not for safety critical applications.”

Instead, Thales is looking at ways to combine expert systems, which can be explained, with the more powerful neural networks and deep learning, which cannot.

“We want to use both approaches, and probably a mix of them so in the end we can have both a high performance and certifiable system, well suited for safety critical applications,” Jourdan says.

Along with the stringent safety requirements, another argument against the swift introduction of AI and other technologies is the effect it will have on employment levels, with staff worried that they could lose their position to a machine or robot.

Those developing AI acknowledge the technology is likely to disrupt the current business models in the mid to long-term, resulting in at least a change in roles for staff and at most a reduction in employee numbers.

But in the short term, AI is mainly being developed to support, not replace, staff, with employees instead performing much higher-level tasks than currently undertaken. AI is capable of providing different solutions to complex problems, each coupled with a number of probabilities, but the technology is not yet able to autonomously decide which solution to select.

Long-term, fewer staff will be needed to carry out certain tasks, but Hall also sees the possibility of new jobs that don’t even exist yet opening up.

“AI audits, data audits and data compliance – there are so many data prominence issues and biases, checking the bias of an algorithm, they’re all types of jobs that need to be done by humans,” she says.

“If we are able to properly introduce AI we could think that in few years time the next ground breaking designs will not be done entirely by humans, but with AI support.”

Giorgio Travaini, Shift2Rail Head of Research & Innovation

This must be facilitated by providing more opportunities for workers to reskill and upskill, Hall adds, but people also need AI training before they even enter the workforce. “We need to get people up to speed with AI. We need anyone at university to have AI awareness courses, we need AI in schools and I want to encourage more kids to do science, technology, engineering and mathematics subjects.”

Travaini sees AI changing business models in other ways, such as finding new ways to optimise research in the industry. He points to rail profiles, which are under constant development.

“You would think that after so many years the rail profile would be the perfect one, but we’re still improving it,” he says. “AI is a very useful tool for allowing the system to evolve faster than we already do. As an example, if we are able to properly introduce AI we could think that in a few years time the next ground breaking designs will not be done entirely by humans, but with AI support.

“It could take one billion different designs and test them all. This is how AI could disrupt businesses. When you’re going to speed up you are going to leave behind anyone who can’t keep up and embrace this technology.”

Regulations and governance

Amid the rapid development of AI and other digital technologies, governmental and institutional responses have often struggled to keep pace with the legal, societal and ethical challenges posed by AI. However, some significant actions have already taken place. In May 2018, the European Commission (EC) issued a communication, Artificial Intelligence for Europe, which sets out a European approach to make the most of the opportunities offered by AI, while also preparing for socio-economic changes and ensuring an appropriate ethical and legal framework.

The EU has allocated €1.5bn for AI investment in 2018-2020, and explicitly acknowledged that transport represents a key sector for the development of AI applications. The EC has also proposed that €2.5bn should be channelled into Europe’s AI capacities through the Digital Europe programme between 2021 and 2027.

The EC has also established a high-level group on AI, including 52 experts from academia, industry and civil society, which will support the implementation of the European strategy on AI.

The European Railway Industry Association (Unife) says autonomous operation is most likely to be improved by AI, but for that to come to fruition two key elements are needed: dedicated investments in the field, together with assessable and certified AI technologies for safety-critical applications.

“Notably, this step would be carried out through new standardised certification processes and the possible creation of extensive open benchmark data sets, as well as the establishment of test fields and tracks for assisted and autonomous driving,” Unife says in its vision paper on digitalisation, which was released in April. “It will also be necessary to produce highly detailed and accurate digital maps of public rail networks, containing standardised data, which are accessible to all stakeholders.”

“In order to allow the commercial use of AI-based products in the transport market, many existing standards and regulations may have to be reviewed and modified. In this context, ensuring the highest possible levels of safety must be a fundamental objective.”

We may still be some years away from the autonomous service that books your ticket, gets you to the station, operates the trains and signalling, and controls everything from fleet maintenance to research and development. But steps are being taken today to bring that dream closer to reality.

Cloud vs Edge computing

A key consideration when developing and installing AI technology is deciding where this processing happens. The debate comes down to the speed of processing against the power needed, with locally-installed computers able to provide an output in seconds. Cloud-based processes are often faster and much more powerful, but can also take longer due to the need to transmit data backwards and forwards between the vehicles, station or maintenance facility and the cloud.

Edge computing refers to data processing power located at the edge of a network instead of in the cloud.

Thales has developed a “3D gate” capable of scanning passengers as they pass through ticketing barriers, which can ensure passes are being used by the pass-holder and not being shared with unauthorised users, and could in future remove the need for any action from a user by using biometrics to detect passengers.

Thales chief technical officer for ground transportation activities, Mr Amaury Jourdan, says that while the system is complex, it must still work within a split second to be viable, which necessitates edge computing.

“In this example it has to be very fast, so cloud computing might not always be the right solution for real time applications compared with edge computing,” he says. “It’s a trade-off and the industry develops both highly-powerful cloud based AI solutions as well as hardware optimised platforms for edge applications. I don’t see this trade-off disappearing, both cloud and edge solutions will find their place in the transport system.”

Watch this video:

Post: Blog2_Post
bottom of page