Security on CAN bus with Ken Tindell

In this episode, we will learn so much about embedded CAN bus, for automobile security and performance straight from the water host, Ken Tindell the CTO of Canis Labs.

A lot of interesting facts about cyber security and automobile hacking that you would not want to miss! Watch through the end and make sure to check the additional resources below.

Listen to Podcast:

Download this episode

Watch this Episode:

Show Highlights:

• Ken Tindell’s background and how he got started with CAN bus and CAN security
• Ken worked with Motorola on designing the MS CAN—the first CAN controller that did all the buffer scheduling correctly
• Ethernet and CAN coexist in autonomous vehicles’ architecture, Ken explains how
• There are two types of major attacks on the CAN bus: attacking the physical wiring and attacking the computer that has access to the wiring. Zac and Ken talk about the mind-blowing advanced techniques of hacking automobiles
• Story of irony. A friend of Ken in automobile cybersecurity had his car stolen
• The biggest car-hacking horror would be through the cloud — cloud-based APIs to phone and at the same time CAN bus protocol hacking on the transceiver pins
• CAN HD, an augmentation for high speed and guarding the CAN bus
• Canis Labs is in the works to provide real solutions for security on CAN bus

Links and Resources:

• Connect with Ken Tindell on LinkedIn
• Visit Canis Automotive Labs' website
• Check out Ken’s Github Page
• Watch related podcast episodes:
  • Better Performance and Enhanced Reliability in the Automotive Electronics industry
  • Electronics Manufacturability and Reliability with QA Guru Cheryl Tulkoff

 

Get Your First Month of Altium Designer® for FREE

Transcript:

Ken Tindell:

So a friend of mine who's in fact in cybersecurity for automotive, had his car stolen.

Zach Peterson:

That's ironic. That's very ironic.

Ken Tindell:

I thought it was a trophy hack from some guy that was just going to leave around the corner or something. But no, it was stolen purely by the odds. In fact, his neighbors had a car stolen the same night. So yeah, in that case, that was an injection attack. And he's been trying to track down how the hell I did it.

Zach Peterson:

Hello everyone and welcome to the Altium OnTrack podcast. I'm your host, Zach Peterson. Today we're talking with Ken Tindell, CTO of Canis Labs. I've been seeing Ken Tindell post quite a bit about CAN bus on LinkedIn lately, and I have done a little bit of research and realized that this guy is quite the expert on CAN bus, specifically in the automotive industry. So I'm very happy to be talking with him today. Ken, thank you so much for joining us.

Ken Tindell:

Ah, good to be here.

Zach Peterson:

Yeah. So maybe since you're a new guest on the podcast, give us a sense of your background and how you got started at Canis Labs and how you became, I guess, such a guru on CAN bus.

Ken Tindell:
Yeah, I started a long, long time ago in the automotive industry. So my PhD was in realtime scheduling theory. Where you have to guarantee everything runs on time. And this was in the mid-90s. And back then the car industry was just moving over to CAN bus and they had a problem with making sure that everything on the CAN bus runs on time. And so Volvo started a project with me. And I started a company to support that project and that's how I got started in CAN bus. That company was later sold to Bosch and then I started at Canis Labs to focus on CAN security because back in the '90s, well, the internet was still dial up and no one had any concept of hacking your car from the internet and things were a bit different now. So that's the new focus.

Zach Peterson:

Yeah, back in the '90s, hacking I think was more associated with a movie, Hackers. Not something that companies have to deal with on a daily basis.

Ken Tindell:

Yeah, yeah, exactly.

Zach Peterson:

So you're really a hardware security firm or a hardware cybersecurity firm it sounds like.

Ken Tindell:

Yeah, well it started in embedded software because that's what we focused on in the early days of CAN. Because we worked with the silicon vendors in the very first CAN controllers that came out. And we focused initially on a security gateway that copies CAN frames from an unsecure side to a secure side, like a firewall for CAN frames. And pretty soon we found that the best way to address security is actually in hardware itself, to try and lock down the ECU that's been compromised in some way and prevent attacks going on.

Zach Peterson:

So you said you worked with the silicon vendors, you were providing IP?

Ken Tindell:

Actually it was how to get the CAN controllers operating according to real time requirements. And the very, very first CAN. Controllers didn't really handle the buffering on the can controller properly. So you'd get a thing called priority inversion where a high priority urgent frame gets stuck behind a low priority frame in the buffer. And then there's a whole bunch of medium priority stuff on the bus that delays the most urgent thing. So everything starts to fall apart for timing. And that's the thing that CAN needs to do right. So we worked with what was then Motorola on designing the MS CAN. It was the first CAN controller that did all the buffer scheduling correctly. And ever since then, every small CAN controller has used the same buffering scheme that we designed for Motorola. So it was less harder IP and more just the intellectual rather than the P.

Zach Peterson:

I see. So this is more design solution for them to then implement in their own hardware.

Ken Tindell:

Yeah, exactly. So they produced the world's first properly realtime compliant CAN controller and then everyone else copied that design afterwards.

Zach Peterson:

So CAN bus has been around for a long time. I know it's decades old.

Ken Tindell:

Yeah '87 I think it was launched. Yeah.

Zach Peterson:

'87. So it's early in the morning for me, so I'm slow on the math. That's 36 years.

Ken Tindell:

Yeah, nearly four decades. That's right.

Zach Peterson:

That's a long time for a protocol to still be the dominant protocol, I would think. I'm having trouble-

Ken Tindell:

Well Ethernet's

Zach Peterson:

Ethernet's older. Sure. But I mean at least the speed is advanced on Ethernet.

Ken Tindell:

Yeah, there's things that with CAN bus. The speeds advancing. So the latest stuff in CAN is CAN XL that goes up to 10 megabits. So yeah, CAN's not stood still. There's all kinds of stuff going on there. And anyway, in real time control systems, it's not about bandwidth, it's about latency. And if you have a hundred megabits network, it's a hundred times faster than the fastest CAN network. CAN still has really short latencies that can't be beaten, because the time taken to get an urgent message on the bus, because of the way it prioritizes it, it jumps ahead of anything that round robin or time division access can provide. And then if you look at some of the ethernet architectures with multiple switches, you've got hop, hop, hop, hop and it struggles to hit those kind of constraints that CAN can meet. And CAN is extraordinarily cheap to implement. So the wires are very low cost. And in silicon terms, you're talking a few thousand gates for a CAN controller. This comes on devices virtually for free in the current silicon sizes.

Zach Peterson:

Everybody at least recently has been talking automotive ethernet, and then for some applications in the vehicle replacing copper with wireless. So probably the most common one was the battery management system for electric vehicles.

Ken Tindell:

Yeah.

Zach Peterson:

Making it a wireless BMS.

Ken Tindell:

Yeah, I looked at battery management wireless some time ago to see if it was possible to do a wireless version of CAN that has those latencies. Because with battery control, if things start to run away, you need to be very, very quick in getting control over that. The trouble is wireless. Yeah, it has some major advantages. Partly not having to get connected to 800 volts of a stack of batteries is a major win there. But I've seen other techniques that basically hop from cell to cell with inductive coupling or magnetic coupling on the physical layer. So you can hop, hop, hop across all of the cells and that you don't have that kind of problem. But that's not my area. But yeah, wireless is solving certain very specific problems there. Of course it also has very specific security problems because if someone can interfere with the wireless, or even worse hack the wireless, as the vehicle's driving along, you have a completely different set of problems on your hands.

Zach Peterson:

Well I bring that up in specifically automotive ethernet, because that has received a reasonable amount of attention in the past as a high data rate replacement for CAN, maybe only in certain part of the vehicle. But as the industry focuses more on autonomous vehicles, there's a lot of data involved with all these different sensor suites and you have to move it somewhere. And you need a high bandwidth protocol to do that. So it seems to me that there's a little bit of a competition here between a more advanced protocol that has higher latency even though it can deliver higher bandwidth versus something like CAN which is lower speed, but has the latency needed for real time control over the automobile.

Ken Tindell:

Yeah. And that's why the designs, they coexist. Because if you're doing realtime control, it's not really about large amounts of data, it's 16 bits for a speed value or eight bits for a temperature value. And you're not sending large data. If you're dealing with cameras and stuff, then obviously that's completely inappropriate for CAN. And that's why Ethernet it's coming in primarily is for the high bandwidth sensors, LIDAR, radar cameras, many, many cameras, and you need a complete architecture for that. Mixing that up with the real time control, particularly the safety critical control, is really a challenge.

So one of the things that CAN does is it's got a property called atomic broadcast. So if you send a frame on CAN bus, it's guaranteed to have been delivered to every other online device on the CAN bus. Because CAN doesn't just throw away a bad frame, it has a low level built into the hardware retransmission. With ethernet, you have a frame check sequence of the CRC. And if that's bad, you just throw the frame away and then you leave it up to a higher layer layer of software to try and work out, did the frame get the noisy on this one? Did that one get it?

And this one not get it. And trying to put the atomic properties of that where you want everything in with a consensus view of a control variable is really challenging on the Ethernet. They were trying to do this here at NASA in the '80s and it's a massive challenge. And when you start to put all the latencies in place for software that says time out, retransmit, acknowledge, did I get acknowledged for everybody? Do I do another round? Blah, blah, blah. Then you are really starting to lose all of that and CAN delivers all of that built into the protocol. So for real time control, it's a different field. And this is why you're seeing designs that are mixed. So there's ethernet for high bandwidth, particularly with cameras, you can't send anything compressed because the artifacts really, really mess up the division systems. So they have to go raw camera images, that's a heck of a lot of bandwidth. And then you're seeing CAN for realtime control and then some gatewaying of data back and forth that is needed to be shared, speed values and stuff like that.

Zach Peterson:

Well, with something like camera images or a radar point cloud or LIDAR data, the architecture that I've seen usually come up is some kind of central processing computer in the vehicle somewhere. And then it connects to these sensors and they're streaming data off at 10 gig Ethernet or higher. I've worked on a system that's dual 28G. Yeah, that's a lot of data and a lot of cameras and a lot of radars all at once. But it seems to me that the solution there, because you have a latency issue, is do all the processing on that sensor. And then send out the results on a faster latency protocol.

Ken Tindell:

Yeah, there's architectures for dealing with that, but that's not what most of the stuff on CAN is. It's things like the indicator lights or the door is open or the locks are closed or the fan in the climate control is moving and the left fan is open and the right fan is off. And they're basically little switches and tiny sensors. So it's a whole different class of problems. So you are dealing with different amounts of data, different types of processing. So they don't really clash. And that's why you're seeing this kind of hierarchy. So there's an ethernet backbone for very, very fast data stuff and then there is CAN for realtime control. And one of the other problems you have is the sensors and actuators are physically where they are in a car.

A door is where a door is and therefore a door lock is where the door is. So these things are difficult to centralize. There is a general trend towards people wanting centralized domain controllers let's call them, where you start to get rid of all these metal boxes of electronic control units and push them into a centralized computer, and then do much more in software and then send end actuator commands back out across a real-time network. And so you've moved lots of the control algorithm into the center. And that's feasible at least, at the initial level, because lots of the control algorithms are actually models built from modeling tools like Simulink and you could just take them out of VCU and push them into the center and then do the control there.

So you take the sensor readings across CAN or whatever, into the algorithms and then back out across the bus. But there's a whole bunch of problems with that. That are technical and non-technical. And so the non-technical ones are you are now changing the whole supply chain from somebody who makes an ECU with a mechatronic solution with the hardware. And now you're saying to give up their algorithms and hand them over as source code or something to somebody else who may be a competitor to do the integration, there's a huge problem there. And then when you're running on a central computer and your algorithm goes wrong, is it your fault? It might be someone else's algorithm ran for too long and now-

Zach Peterson:

Oh, meaning the data that they-

Ken Tindell:

...liabilities.

Zach Peterson:

You mean the data that they gave you could have been incorrect or the result that-

Ken Tindell:

No, it's more their software could have gone crazy and used the load of CPU time and then you didn't get a look in because the scheduler couldn't fit you in, because their software ran too long. Because now you're on a shared CPU resource. So now you've got to worry about who else is on the same CPU. And then if they have a supply chain attack, so someone got some malware into their component, but it made your software component fail, who's at fault now? And even if it's easy to answer that question, it's not easy to answer the question, prove it.

Zach Peterson:

Oh sure. Sure. Yeah. Especially when the supply chain is not consolidated in a way that would enable that very clearly.

Ken Tindell:

Exactly. Exactly. So the whole idea is you're taking suppliers that we're delivering you mechatronic solutions and now they're supposed to be delivering you little bit of tronics. And somebody else maybe is doing the software or their software's being run in an environment. And now you've spread around the supply chain liabilities and there's clear lines of responsibility. So it's some challenges there. You can solve these, but it's not as plain sailing as drawing a square box on a PowerPoint diagram and then just saying software.

Zach Peterson:

But I guess solving those types of problems and yet still having separate companies and separate structures that requires coming together in these kinds of partnerships to develop these systems.

Ken Tindell:

Yeah, yeah, absolutely. And I've seen different silicon solutions. So I was at a show where one vendor was showing effectively you could virtualize ECUs, because they had so many CPU calls and such strong partitioning and memory between them and the internal arbitration between the AXI buses on their crossbar and stuff was done such that you could completely isolate these separate cores. And then you're dealing with a solution that if one core goes crazy, it can't disrupt somebody else's core. And at that level is you have virtual electronic control units. And then that hardware solution is a way of addressing that problem. And you could throw hardware at this and some people are looking to do that.

Zach Peterson:

Very interesting. So going back to security for just a moment. So one of the main reasons I wanted to bring you on the show is because I do see you post a lot about CAN bus security and I think we have this idea that sometimes when the protocol is lower level, maybe it's a slower protocol like CAN, at least in terms of the data rates, it is not going to be something that is worth hacking, that's not where the gold is. But I think that's totally wrong and you've shown some examples of it. And we actually did a podcast with Joe Grand, who's well known as a hardware hacker. And he had mentioned some points about getting physical access to even these low level buses that could allow you to totally take over a system. So what exactly do you do in the area of security? And maybe we can talk about what the implications are of that.

Ken Tindell:

Yeah, to be clear there are two major types of attack on the CAN bus. One is when the bad guy has access to the physical wiring. And then the other is where the bad guy has access to the computer that has access to the wiring. You see what I mean? So for example, if you have a tire pressure monitoring system, they talk to sensors over RF. I've seen hacking of basically spoofing those tire pressure things to cause a buffer overrun to get control of the ECU. So you could access the tire pressure ECU from outside the car by lying to it about the tire pressure data and spoofing it, and then you're on the CAN bus because you now have access to an ECU that's connected to the CAN bus.

Zach Peterson:

So what could you do with that? Is this we're going to pop the locks and start the engine?

Ken Tindell:

Yep. Yeah, you could do that. Yep.

Zach Peterson:

That's it. Yeah,

Ken Tindell:

And then there's not normally one CAN bus. There's normally several can buses, at least two, and then in higher end vehicles, 3, 4, 5, 6, 7, 8 at different CAN buses, they're partitioned off because they're running out of bandwidth and they need to run these buses in parallel. So if you get access to the powertrain side, then you can spoof CAN frames as if from any other ECU on the powertrain side. So you can do all the things that the powertrain system could do. So that's the gear box. Typically the brakes, the engine, often headlights and things like that are on powertrain because of physically where they are. So you could spoof anything.

Zach Peterson:

So that's speed up the vehicle to a hundred miles an hour, slam on the brakes.

Ken Tindell:

So it depends. So there's a system called E-Gas, which is basically electronic throttle and no physical linkage to a plate on a spring that opens the air input to an engine. Then you can race the engine and if you have access to the gearbox, you could potentially move the gear. So for example, the gear shift is often on CAN. And when you are doing that, really you're just activating a switch and it sends a frame to the gearbox to say please go into this gear. And then the whole gearbox powertrain system then accepts your commands or rejects it if it's too dangerous to shift gear and so on.

Zach Peterson:

And here I am shifting gears thinking that I'm actually shifting actual gears.

Ken Tindell:

Yeah, no, no, not anymore. Certainly not in an automatic gear boxes. Even all these clutchless systems, they're again, they're physical gear boxes, but the software has asked for the gear. So if you spoof those frames, you could put the current into a certain gear, you could erase the engine, you could cut the engine off. Things like power assistance on the steering, the electrical power assistance, you could tighten up steering. So it's very hard to turn or you just disable it.

Zach Peterson:

Wow.

Ken Tindell:

There's all kinds of stuff that you could do. And then if you could get inside the ECU itself, like for example, you could hop from one ECU to another by exploiting another buffer overflow and then jump into that ECU, your access directly to the actuators. So no longer just responding to certain message requests, you're now directly controlling actuators. So you could do all kinds of stuff.

Zach Peterson:

So when you say overflow the buffer, this is just bombarding with messages in a format that you know will trigger a system the way you want it to? I think that's a little bit [inaudible 00:20:53] vague.

Ken Tindell:

 Yeah, there's nothing new about this. This is classic how you hack embedded devices, is the quickest way to do it is to pop the firmware out and then run it through the Ghidra tool that the NSA has published, which has got this a reverse engineering tool and gives you C that represents the code it's seen. And then you can run it over some tools, look for buffer overflow bugs, and then you know the messages to craft to get it to fail.

Or the slow and simple approaches to bombard it with just random garbage and see if it crashes. And as soon as you get a crash you know that something in the stack got corrupted or something. And then you start to hone in on exactly what caused that crash. That's how the tire pressure one is done, is you just produce garbage sensor values and then it crashes and you go hah. And then you hone in on what caused it to crash and then you get a magic payload to take control of ECU. But that that's how all of this hacking stuff works in a large part.

Zach Peterson:

So for CAN bus it is copper wire. And if you were going to jump directly onto the CAN bus, you have to get physical access somehow. How are people doing this? Aside from just drilling a hole in the side of the car.

Ken Tindell:

Yeah, so this is a wired access to the CAN bus and yeah, you just need access to that twisted pair. You can cut it with a pair of wire cutters and splice in that little magic box. And then that magic box sends CAN frames on the bus, but also because it's directly on the wires, it can start to screw around with the voltage levels and stuffs to make it much harder for the system to respond to an attack. So the way you find the connectors is yeah, the wiring diagrams tell you where the wires are. And you know the ECUs and what they do. And you say, right, I need access to the powertrain ECU. So for example, one of the cars we're looking at, the engine immobilizer ECUs on the powertrain side. So are the headlights. So you crack open the headlights and you pull the connector off the back, plug into that, and now you can directly send commands to the engine immobilizer saying, "Hey, I'm the key fob, and I say it's okay to drive." And it's like, "Okay."

Zach Peterson:

Wow. Okay. So basically any two devices or any two subsystems that are on the same bus, the one that's easiest to access physically, that's the one they go after. And then they can get access to every other device on the bus.

Ken Tindell:

So you can just send messages. You just spoof it and you lie and pretend you're on the other one and it just accepts your messages. So the headlight one's quite good because even once you physically moved the headlight you haven't done too much damage and then you can put the connector back and everything, and sell the car on. The ones where you punch a hole in the side of the door or in the side of the wheel arch or something, you've obviously got some damage there. So these things are sold very cheaply in export. They drive them away, they cut off any tracking antennas, they stick them in a shipping container and then they ship them abroad and they get bought for a small amount of money and everyone's making money out of this except the insurance companies and the owners.

Zach Peterson:

So you had mentioned just a moment ago that they can figure this out from a wiring diagram. Would this be the wiring diagram from a control from a repair manual?

Ken Tindell:

Yeah. Yeah. So if you look at a wiring diagram, all the CAN buses are labeled and they normally got colors for CAN high and CAN low and the twisted pair. And they label up the connectors and they say this is. Because the mechanics have got to go in and repair these things. So they know where the wires are going to work and then they'll get a stolen car and then they'll have a hacking of an ECU in there to see if they can spoof it. And then once that information is known, they make a little tool that is about $10 bomb cost. And then on the dark web you can sell that for $2,000, $3,000, $4,000.

So you see them sometimes on websites, what's it called? Backup keyless override device or something like that. And it's to basically avoid all the immobilizer stuff. And the one I've been looking at in a little JBL Bluetooth speaker thing about that big. So if you get stopped by the police, you say, "Hey, it's a Bluetooth speaker." You just disguised it and even go with some kit that they can see. Yeah, so key for the attackers there is they have knowledge of the things. They've done reverse engineering and they know how to package up an attack that seems sophisticated if you take a look at it. But these dumb thieves, they just buy a box and they press the play button and it deactivates the immobilizer and opens the doors.

Zach Peterson:

That's incredible. I guess I shouldn't be surprised that as the technology in cars has advanced, the methods for stealing them or taking control of automobiles also have had to become more advanced. But I didn't think it would've gotten to the point where you can just go on the dark web and buy a little box that overrides a vehicle and lets you get in.

Ken Tindell:

Yeah, we've been here before with keyless entry and relay attacks. They're quite famous now and the manufacturers know about this much more. And that's a very sophisticated attack. You've got wireless antennas trying to power up a key that someone's left by the door and then the relay thing. So it's actually a very sophisticated attack in terms of the kit. But the thieves don't have any level of sophistication. They just wander around pointing it around until it gets a hit. So CAN injection is the new relay attacks. And it's in many ways much easier to mount those attacks than the relay ones, particularly as manufacturers have started to respond with keys that go to sleep when they're not moved and things like that.

Zach Peterson:

Interesting, interesting.

Ken Tindell:

So that's the common technique for it now.

Zach Peterson:

So you said that the manufacturers know about the relay type of attack.

Ken Tindell:

Yeah, yeah, that's quite common.

Zach Peterson:

So are they aware of these CAN bus type of attacks, varying degrees?

Ken Tindell:

Well, it's a bit like when an ant crawls over your toe, have you noticed the ant or is your toe... These are gigantic corporations and the knowledge of this space is not uniformly allocated around the company. So I'm sure lots of the manufacturers know there are people that know about these things, but is it common knowledge like it is for relay attacks? I don't think so. A friend of mine who's in fact in cybersecurity for automotive had his car stolen.

Zach Peterson:

That's ironic. That's very ironic.

Ken Tindell:

I thought it was a trophy hack from some guy that was just going to leave around the corner or something. But no, it was stolen purely by the odds. In fact, his neighbors had a car stole on the same night. So in that case that was a CAN injection attack. And he's been trying to track down how the hell they did it. And The Times newspaper picked it up and ran the story and the manufacturers quote was like, "We recommend you put your keys in a metal box so they can't be used to steal your car. And it's-

Zach Peterson:

Really? That's their solution?

Ken Tindell:

But that's not how it works. That's for a real relay attack. So it was quite clear they're talking to the journalists, they're talking about CAN injection and it's just the same old same old spokesman stuff. So it's not culturally through the car companies, yeah, I think.

Zach Peterson:

Wow. Okay. So what is the solution, at least on the system side or on the design side? Is there something that you add to the PCB? Is there silicon IP that has to get added into the chips that are being used in these different systems to then access the CAN bus?

Ken Tindell:

Okay, so the first thing to point out is a lot of these attacks on new cars, those new cars, they've been in manufacture for two or three years now. They started life maybe eight years ago. And eight years ago this sort of space was a very different world that we were looking at. So a lot of the security in those cars is not like they're doing at the moment. So one of the solutions to this is to have encryption and authentication of CAN frames. It's a bit of a pain to do it for all of them, but certain key ones to do with things like the immobilizer and the locks and stuff, you can put encryption codes, authentication codes inside the CAN frames. And then if you are trying to hack it, you can't spoof anything because you can't get the encryption codes because they're unique per vehicle. And so that won't work and you just made a hole in the vehicle for no purpose.

Zach Peterson:

So this wouldn't be a case of getting the firmware and being able to reverse engineer it and see this, this would be hard coded into the silicon.

Ken Tindell:

No. So you can do it in software/ and in fact my company, we've got a little security library that's about two kilobytes of code. It really small. It uses AES and proper encryption. And as long as you burn different keys into different cars so that all the keys are unique, there's no one CAN hack box that they can get. And they need to get the keys out. Now getting the keys out is not getting at the wires. You've got to get into the CU itself. You've got to take the ECU out, you've got to put it on a bench top, get a JTAG debugger or something on there, maybe one of these differential power attack things that look at the voltage power lines and can work out states of transistors and stuff. No thief's going to do that outside of the road.

Zach Peterson:

They're trying to just get in, get the car and then get gone. They're not going to take the time to actually extract an ECU, go back to the lab, spend a few days and then come back and say, "I'll steal this car now."

Ken Tindell:

Yeah, exactly. So I mean you do have to be a little bit careful because if you're not careful with the protocol for communications, because you need effectively a rolling code. And one of the new standards for encryption o CAN has a rolling code of just eight bits. So you only need to see 256 frames and you've seen all of the combos. And therefore if you just captured them all, you could replay those messages back and they'd look like they came from an authenticated device. So the model there then, the thief would attach something to the CAN bus, leave it, it would then listen to the key being unlocked. If you remember back in the old days with that remote fobs, they would have radio receivers in car parks and they'd listen to your car being locked or unlocked, and then they'd come back and then replay the radio signal.

So it's similar. You could do that. So you'd to be careful. But if you are sufficiently careful, you could do a firmware update of these ECUs and then the car wouldn't be stealable via this CAN mechanism. So that's if you're like the wired street thief attack. But the other types of attack is where you break into the ECU itself. And encryption is less helpful there because you can get the ECU to do the spoofing and it's got access to its own keys and so on. And then it's much easier for it to exfiltrate those keys to you or just spoof the messages you wanted anyway because you've got the encryption keys in there. And then that's where you start to need to look at proper hardware solutions that lock away an ECU that's failed in some way and has been hacked. You need to make sure it can be cut off the bus and denied access to everything else. And there there's a whole bunch of those techniques.

Zach Peterson:

This is really interesting because when something like the relay attack becomes common knowledge, suddenly all the car makers, it seems like, have come together and said, "Hey, this is how we stop this and it's in our collective interests to stop this kind of attack." And now that everyone knows about it, there are things that can be done. But when we're talking about something like CAN bus attacks, I'm just wondering what does it take for the automakers to do something? I don't even know what the something would be, but to do something. If you're talking about someone taking over a vehicle and then being able to control a vehicle to the point where they could even cause an accident, that seems like a lot of liability on the shoulders of automakers. Is it just a cost versus liability calculation?

Ken Tindell:

I don't think it's as cynical as that. It's a very complicated space. So you hijack an ECU through a firmware bug/ you can do a lot of things on the CAN bus/ and because you have direct access to the two GPIO pins into the CAN transceiver, you don't even need to use the CAN controller that's on the chip. You can actually go underneath the CAN controller and then just bit bang those pins and screw with the CAN protocol itself. And I've got a hacking toolkit that's open sourced to prove some of these things. So you can do all kinds of stuff. And so defending against that is a different class of problem from just using encryption, because encryption is like a software layer above CAN.

But if you can come in and start screwing around with the actual underlying protocol, yeah, it's a different level of defenses you need to protect from an ECU that's gone bad. You shouldn't have an ECU bug, but I think we all accept that this kind of software has bugs in it that could be exploited and do awful things. So it's not just in the future, it's not just theft, it's also attacking vehicles for ransomware for example. You can imagine a very large payoff and hence a very, very strong incentive for someone to try and do that. I've seen some research on detonating airbags over CAN bus.

Zach Peterson:

Really? Wow.

Ken Tindell:

Yes. Because it turns out that in fact it's a requirement in most countries to detonate all the airbags when you go to recycle the car. Because well they're like hand grenade detonators, they're little bombs to inflate the airbags really quickly. And then you don't want your guy in a scrapyard having his arm taken off, lose a hand to one of these things as he's crushing the car. So they all must be deactivated. And the way you do that is you hook up a diagnostic tester thing in a workshop and then you just go around, you air horn, everyone get out and then you just detonate the airbags in sequence. But they were able to show that the safety responses there of the vehicle has to be stationary and you have to be your diagnostic mode. The way it finds out whether that's really the case is it's looking at CAN bus to see what the vehicle speed is.

And of course you just lie. And then send, "I am a tester, honest." Commands. So you lie to the airbag ECU and say that you're moving, you're not moving, and the diagnostic tester is attached. So yeah, if you were able to hijack an E C U, you could embed that inside the ECU itself and then you're looking at all these different vectors in there. And to try and deal with that is a much, much more complicated picture than just encryption on the CAN bus.

So the street theft I think is quite easy to solve. And lots of manufacturers in their current projects are already doing things like encryption on the CAN bus. What they really need to do I think is to put a firmware update out for everyone who's got the cars already that were made and designed some time ago. And that seems a bit unfair to leave people high and dry when you can fix it now, but you can fix it retrospectively because it's a software change. But how you address some of these wider issues on access to these actuators and things is a different matter. Yeah, it's a big issue.

Zach Peterson:

So the companies, I guess, the OEMs or whoever they've contracted out their designs to, the actual physical design of some of these modules where you could get access to the bus, are they designed with any level of physical layer protection built in? Or is that also an afterthought? Or is that an issue that adds cost and size and weights. And obviously with vehicles we would like to have weight lower, not higher, especially today's energy efficiency focused world. So are those kinds of things just not even a thought anymore? Or are they pointless because if I can access two wires on a bus and get access that way, it doesn't matter what I do on the PCB or in the assembly, I can get access to the wires.

Ken Tindell:

Well, it's both those things. So the wired one, they're fundamentally it's extremely difficult if someone's got physical access to the car to stop a bunch of these attacks. But the one that's really scary is if you are cloud hosted and there's a telematics API that your phone uses. They didn't secure that end point and now you can push 10,000 little malware deployment things over the infotainment or the telematics gateway or however it's done, into ECUs, all at the same time. And then detonate their airbags all at the same time or just cut the engines or whatever, all at the same time. So if you were looking at nation state level of hacking, you can turn a fleet of cars into an area denial weapon if you like.

Zach Peterson:

Oh Lord.

Ken Tindell:

Just shut down all the highway network. So that's the, I won't say paranoid, but if you do your flow charts of what could go wrong, that is at the maximum end of the appalling things that you could do if you did that. And so there's a very strong incentive for a bunch of well-funded groups to do that kind of thing. So the manufacturers have to respond obviously to that. And it's difficult because it crosses so many domains. So cloud-based APIs to phone apps, who's an expert in that? And at the same time CAN bus protocol hacking on the transceiver pins. It's a huge disparate area. And trying to wrestle with that whole domain is very, very difficult.

Zach Peterson:

Yeah, it almost seems like you have to have people collaborating from across these different disciplines within a manufacturer and a third party, whoever is hosting that cloud endpoint to then capture and process that data.

Ken Tindell:

Yeah. Well, it's really difficult because for example, you see the OBD-II connector on cars is used and you have these little dongles the insurance company makes you put in there, then it monitors your driving style to then decide or charge per mile or charge did you break sharply a lot. And they'll charge you for that, which seems to be a stupidly bad idea. It's like, "Oh, I just ran this person over because I didn't want to be charged $5 by breaking sharply." It's really bad idea. Anyway, the electronics and those and the firmware in those and the cloud services for those, nobody's really doing a proper control of that.

Because if someone got into the servers, which is not that difficult, particularly if they're not looking closely, You have to be top of the line internet cloud hosting company to really have the expertise to handle that. And then there's firmware on the dongle ECU, or dongle device, and it's got a little micro controller in there and someone's put the firmware together. And you know how these things are built. They normally just  and they grab the first thing that works and they glue it together and it looks like it works and they ship it. And in the case of these insurance company things, very often it's a white label thing. So it's like some company has got solutions and then they badge it up for the different insurance companies.

Zach Peterson:

Yeah, they just slap the insurance company's brand on it and call it a day.

Ken Tindell:

And ironically, the insurance companies unable to assess the risk for these things. They're putting it off.

Zach Peterson:

Yeah, talk about irony.
 

Ken Tindell:

Yeah. And then the really bad part of it is there is no proper post forensics analysis going on in this. So you see a lot of crash analysis for the police looking. And each individual ECU very often has its own data capture around an accident or something so that it can work out what the hell is going on and know if it's where the liabilities are, not mine or mine. But trying to put all of that together in a big picture, it's actually very difficult for a vehicle forensics guys to collect all that data and then build a proper picture. And if it's not being captured properly. So if for example, this third party dongle was plugged into OBD-II and it did something terrible on the CAN bus, how'd you even prove that after an accident? No one.

Zach Peterson:

That's a great question.

Ken Tindell:

Sorry?

Zach Peterson:

I said that's a great question. How do you even prove any of that? Forensic investigation is behind the times, I guess, when it comes to some of this stuff that could happen on a vehicle. It's one thing with looking at someone's PC because you can get access to all the data on a PC, but cars are rolling computers these days. How do you get access to any of that. And it doesn't seem to me that any of that is really being logged anywhere unless there is a cloud connection to some third party application.

Ken Tindell:

And then that would be a security weakness into the car from the other way.
 

Zach Peterson:

Yeah, exactly. Yeah.

Ken Tindell:

It's not easy then. It's a very complicated multi-dimensional problem here. Yeah, so intrusion detection is one of the techniques that people are looking at. So intrusion detection is the CCTV of embedded security. It's like, well, we couldn't stop anything, but at least we could see what was happening. And there was these bunch of events and stuff. And so we saw three attempts at breaking into the car through these techniques and we saw these kinds of things happening on the bus. We saw this. And then you capture that, store it, and then if it's a stolen vehicle, you recover it later maybe. And then somewhere, somehow working out what they're doing has been captured and then you know.

Because otherwise all is your car just went missing and you don't know how they did it. So intrusion detection is part of that forensics of post-incident analysis. And there's work in all of these areas. And it's just a very, very complicated problem. If you think about it's probably one of the most complex security domains that crosses from right down at the tiny wiring level and microcontrollers and firmware all the way up to cloud hosted services. And the whole shooting match, it has to be secure. That's really, really a challenge.

Zach Peterson:

Well, at least with access to the physical layer, where you're talking about injecting CAN frames onto a bus to try and overwhelm a device and cause it to crash, or to get it to read the frames that you want and act in a certain way. How can you, I guess, through software or through hardware, find and sniff out those malicious frames and then prevent them from getting onto the bus? You had mentioned like a firewall technique earlier. What does that look like? Is that just a chip that gets added onto the design or is it something that gets implemented in silicon?

Ken Tindell:

Yeah, so that's actually something I'm working on right now is a chip. Because you don't really want to do it in firmware inside and ECU, it's got all this other stuff going on because then you've just basically broken the firewall. So it'd be really, really useful if it's an external chip that you fire stuff in and stuff comes out if it's allowed. And one of the things that we've been working on for a while is a technique, we call it CAN HD, it's an augmentation for high speed and guarding the can bus. And it injects out of band data that sits in between the sample points in a CAN. Because CAN is quite slow, 2000 nanoseconds between two microseconds between CAN sample points. And you can put a whole bunch of edges and bits in between some of those sample points under certain rules.

And that lets you tag things like where physically the CAN frame came from. So if your data's coming out of your CAN controller that's on the microcontroller, and if it passes through, it can get tagged up with where it came from. And then a centralized, not just an IDS intrusion detection, but intrusion detection prevention system, can decode those tags in real time as they come past and then know that that frame came physically from where it shouldn't. "So I'm the immobilizer key fob, please deactivate." It's like, "No, you're not. You're the entertainment system. So I'm not going to trust that and in fact I'm going to destroy that CAN frame before anyone gets to see it. And then I'm going to send a message back to the little tiny piece of silicone that's done that fingerprinting and say switch this guy off because he's been compromised. Don't let it talk to the rest of the bus now." So that's what we worked on. In fact, I've got a little tiny prototype here.

Zach Peterson:

Yeah, let's see it. Don't drop it. There we go.

Ken Tindell:

So that that's designed to as match the pin out of a fairly common CAN transceiver. So this is an integrated security CAN transceiver. So on there there's a little tiny FPGA with our harder IP in there, a crystal and a CAN transceiver and a resistor I think. And the little dots on the side are just a tear off tab for the tag connect programming to probe. So once you've programmed the FPGA app, you can tear off the tag and then that becomes a multichip module that's a drop-in replacement on a PCB for what would be a fairly conventional CAN transceiver.

So it pretends to be a CAN transceiver, but it's got all the logic inside. That's actually probably the best place to do this. So NXP have a CAN transceiver called the Stinger, which has a very simple white list of CAN frames that you are allowed to send. And then if you send a CAN frame that's not on the list, it destroys the CAN frame. And if someone else on the bus sends a CAN frame that's on your list, then they must be trying to spoof you and it destroys the CAN frame as well.

So I see that's a nice way of doing it. That doesn't work for wired attacks because this whole destroying thing relies on the CAN dominant bit being able to pull down because CAN floats to recessive if you like. And then anyone can pull the line into the dominant state. And hackers can drive a recessive actively onto the wires because they've got physical access to the wires. So it doesn't work on some attacks, but it certainly will stop another ECU that's that's been hijacked from doing anything because that hasn't changed the electronics to drive the bus recessive, like I said. So there are hardware solutions to this you can put onto the PCB. That was designed as a drop-in retrofit, but to cover the long term is actually built into the can controller logic itself. So that's just part of the CAN controller.

Zach Peterson:

So a hardware solution like this, I'm sure if you scale up to mass production, obviously costs are going to go down. Are the OEMs receptive to that kind of solution? Just a little thing that they can throw on the PCB. It's easy to design around and it carries very low cost and no weight.

Ken Tindell:

Yeah, I see quite a lot of resistance to this. NXP I think are suffering the same reluctance to use a piece of hardware. And you see this across the whole tech industry, there's this dominant feeling that encryption is the way to solve a security problem, that all security problems are encryption and encryption solves all security problems. And then once we do that, we don't have any security problems. And it solves your authentication problem, but what it doesn't solve is your denial of service attack problem. So for sure, you send these messages and they're all authenticated, but if they never get through, what happens then? One of the car projects I'm working on, the shift switch lever, in fact it's a button press you press to go to drive, that send a CAN frame every so often to say, "Hey, I'm alive, I'm alive, I'm alive."

And then if the gearbox controller doesn't hear from it for a period of time, it says it's bust. And the shift controls are a vital part of the car. And so the failure mode is to say, okay, so the car has had a catastrophic failure of some kind because this is a vital piece of a kit and we will slow the car down, applying the brakes if necessary until it's stopped. We'll put on the parking brake and then the car won't be allowed to move again. And you have to call out roadside rescue.

So if your goal was to screw up a car and make it stop itself alongside the road, all you have to do is just block those CAN frames from that heartbeat CAN frame, kill it as it comes onto the bus. And no matter what encryption you're doing, it's not going to solve that problem because the messages aren't getting through. So then you've brought that car to a halt. So if your attack was on the basis of demanding ransom for stopping cars on highways and then ramping up the number of cars until eventually they pay up, it's effective technique.

Yeah. So you do need these hardware security things, I think, to block those kinds of attacks. But we haven't seen those attacks yet. At least they haven't been made public If those attacks have been going on. They're not the sort of attacks you would make public, are they? =

Zach Peterson:

Well, cybersecurity issues have, I don't want to say dominated the news, but there was a period of time where it seemed like there was a hack being reported on some news channel every week. And it was big name brands that everyone knows about. It's just that they keep making the news because they're places that retail consumers use or purchase from. And it was the online account that everybody accesses and just expects to be perfectly secure. So it seems to me that anything involving hardware security just gets swamped and is a backpage story compared to target lost 50 million customer data profiles to a hacker, something crazy like that. Because those are actual news stories that happen.

Ken Tindell:

So I think there's an asymmetry on the attitude to proper hardware security. Because the thing is that the hardware security is actually easier to do. Because whenever you have encryption, a friend of mine said basically if you have an encryption problem, if have an encryption solution, you've just created a key distribution problem or a key manager problem. So all these cars have to have unique keys, ideally they need to be changed to prevent certain kinds of things where you can over time work out what the keys are. So they have to be updated at every service interval with new keys. And then somewhere someone has to have a database that manages and looks after all the keys so that when the workshop plugs in a tester, it says okay, that's serial, blah, blah, blah, blah, blah, then blah, blah. Here are your keys.

And then if you take a spare part and program it in, you've now got to program that with the keys for that vehicle. And if you take a spare part out of a vehicle, you've got to reprogram it to use it in another vehicle and da, da, da, da. And then someone money breaks into your central database because that's what happens in central databases. And then they exfiltrate the entire database, which would only be a few megabytes probably for all of the vehicles. And now all the keys are known because someone stole the key database, because someone left a USB stick containing it on a train or something. So you have lots and lots of secondary and tertiary problems with encryption. So it doesn't solve all the problems. And you have to really, really get it right to manage all this stuff.

You see people, people's Bitcoin wallets being stolen all the time, that's an encryption. Those keys are supposed to be looked after. And people don't. They stick it on a USB stick. So the hardware security's got a very nice model of deployment. It's physically there. I don't need to do anything. It's just there. It's part of the design. And then it works. And there's no need to change it and USB sticks don't cause it any problems. But these are quite complicated issues to talk through in a very, very complicated space. And I think that's why these things don't get the focus they should.

Zach Peterson:

Yeah. Yeah. Well, we've only got a couple minutes left, but I think for just the final question I'd ask you, what's coming on the horizon for your company?

Ken Tindell:

Yeah, so our focus at the moment is a secure CAN controller that actually has a whole bunch of these features for denial of service problems, saying you're not allowed to send that frame. Or you are allowed to send that frame, but you can't send it that often. And then we'll have an external CAN controller. And because of that, nothing on the host can get access to the pins on the CAN transceiver. And then this really starts to push the security. So a little external can controller chip, it's a harder IP, and if we make that into an ASIC or high volume chip, then we have a real solution for this security on CAN bus.

Zach Peterson:

Very cool. Very cool. Well, hopefully as some of these things develop, we could have you come back on the podcast and discuss all this with us.

Ken Tindell:

Yeah. Yeah.

Zach Peterson:

Well Ken, thank you so much for joining us. We've been talking with Ken Tindell, CTO of Canis Labs. To everyone that's out there watching, make sure to check out the show notes. We're going to have some great links there so that you can learn more about what Canis Labs does. And of course, make sure to subscribe to our YouTube channel. You'll be able to keep up with all episodes and tutorials as they come out. And last but not least, don't stop learning. Stay on track and we'll see you next time.