Reverse Engineering: Is it Worth it?

Pierce Design’s Ethan Pierce will share his insight into reverse engineering with PCEA. We will also discuss firmware reverse engineering.

Listen to the Podcast:

Download this episode.

Watch the Podcast:

Show Highlights:

• Introduction to Ethan Pierce and a quick preview of his upcoming webinar with the PCEA regarding reverse-engineering
• How to acquire reverse-engineering skill sets?
• Retrofitting a system versus creating a net new ecosystem of products
• Is reverse engineering cost-effective?
• Ethan advises designers to keep records and documentation as much as possible. “take as many pictures, take as many pictures, photos, notes.”

Links and Resources:

• Connect with Ethan Pierce on LinkedIn
• Visit Pierce Design's website
• Register for Ethan Pierce's Webinar: Reverse Engineering PCBs: How to Recreate a Lost Design

 

Get Your First Month of Altium Designer® for FREE

 

Transcription:

Ethan Pierce:

I used to think that my father was nuts for keeping around all these old SGI machines and old Dell. And they end up becoming so invaluable because just to be able to boot it up, open that CAD package from 2000 and whatever, load up that native file and look at that data, dude, it becomes priceless.

Zach Peterson:

Hello everyone and welcome to the Altium OnTrack podcast. I'm your host, Zach Peterson. Today, we're talking with Ethan Pierce, Director of Hardware Engineering for Pierce Design. Ethan will be doing a webinar soon through PCEA and it's on a very interesting topic related to reverse-engineering of printed circuit assemblies. And I'm very excited to have him talk about this topic with us, as well as get to know Ethan quite a bit more.
Ethan, thanks for joining us today.
 

Ethan Pierce:

Thanks, Zach. I'm really excited to be here chatting with you guys.
 

Zach Peterson:

Yeah. I have to admit, before I saw some of the promo material on LinkedIn for the webinar that you're doing, I didn't know who you were. But I'm glad to actually ... I'm always glad to meet new people in the industry. And the topic that you'll be talking about is pretty interesting. Maybe you could give us just a quick preview.

Ethan Pierce:

Yeah, absolutely. So, one of the things that when I went to PCB West and I started getting involved in the design community as a hardware engineer, I noticed that one of the things that kept coming up across my career was retrofits or existing systems, and needing to interface either new technology with it or being able to just keep the darn thing up and running in order to continue to operate a business.
And what I wanted to do was instead of just kind of giving a high-level talk on, "Hey, this is how this stuff kind of works. You'll figure it out, it'll be great." I wanted to give the community a set of tools and processes so that they can use their traditional engineering thinking and have a structured way that they can take a printed circuit board assembly, apply that logic and thinking in order to recreate something.
Typically, when somebody first goes into reverse-engineering, nobody teaches them how to do it. And it's this wild ride of, "How do I create schematic symbols?" It's forgetting their basic ... The things on how to create a board, it's kind of reaffirming and setting up that structure so there's a clear pathway to recreate that dataset.

Zach Peterson:

The most common instance that I have seen or at least that I've been asked about in terms of recreating design data or recreating an assembly from fragments of the production data let's say or from just a PCBA is someone has Gerbers. They don't have the native files or the native files are in some old CAD program so they're not usable in whatever the modern tools are. And then they want to recreate the layout from those Gerbers.
I've seen that come across comments on some of the YouTube videos for Altium Academy. I've been asked it directly on LinkedIn. I've seen it in comments on posts on LinkedIn. I started noticing that and I realized it was a pretty common issue.
So, when you say recreate, are you talking about just from fabrication data? Are you talking about from a PCB assembled, the whole gamut? Give us a little more insight on this.

Ethan Pierce:

Sure. Going deeper into it, the idea was to start with, okay, I have a circuit board. I don't even have Gerbers or production data and recreating that thing from nothing. And so, I think that there's some value there and just using your basic technician skillsets and applying them to engineering to recreate the dataset.
But I think that there's a stone to be uncovered that I do want to touch on in that talk about taking that production data and then recreating it. When you work with a contract manufacturer, you're working with a design shop. Sometimes they want to withhold the native files or for whatever reason, you only have the production files.
And I think part of the game is taking that file type and getting it in like just wrestling with that dataset and getting it into something usable. I think that that's something I'd like to really get deeper on. If not for this talk with reverse-engineering, a subsequent one, this is something that I want to continue to invest in for the community. Because I do think that there's something there of, "I've got a Gerber dataset, it's effectively polygons. And I have this EDA tool. I can label nets and I can label pores. But how do I then bring that in and convert it?"
So, that is something that I do want to get deeper on, but I don't have an established workflow yet in how to bring Gerber data into an EDA CAD tool where you can just highlight something, "Oh, that's my ground pore. Oh, that's my 3V3."

Zach Peterson:

Yeah. Because in Gerbers, I mean it's all visual. You have to make sure that you're catching those connections especially when it's between layers, between all those different pieces of copper in order to make sure that you recreate them correctly in the schematic and then eventually in the PCB layout.

Ethan Pierce:

Oh yeah.

Zach Peterson:

And I think when some people start with the Gerbers, they're just thinking, "I don't even care about the schematic. I just want to know how to get to the layout." It's like the logical connections kind of go out the window.
 

Ethan Pierce:

Right. I mean, you know this, Zach. There's so much data and there's so much of a story that's told from the layout in terms of the copper pores. Or even when you've got some signal trace, somebody has to do some dance in a low-layer count board in order to keep a return path underneath a trace, but they're jammed in there, so via down, via over. And they've got a ground trace, a ground pore right underneath.
So, there's so much of that story. And right before you and I jumped on, I was thinking this through. It's like you've got a design that's already done. There's already been hundreds or thousands of engineering and operations and business hours to make that thing exist. And now you've got to go and you've got to recreate that product from scratch. And that's an unknown thing to do.
But knowing that, okay, there's a story, there's not just a layout, there's a schematic, there were product requirements and having to go through and build back up the stack from I have a physical thing, whether that's a physical thing or I have a dataset and building that back up into a product, it's an interesting process. And I think it's something that's really valuable for engineers and designers to have.
Yeah. I think it's something for all engineers to be able to know because it's definitely not something they teach in school. Definitely not something you're going to have in your junior year. But when you're mid-level or senior engineer, somebody's going to come to you and say, "Hey, we've got this multi-million-dollar system running. And there's this board. And we only have three or five of them. But if this thing goes down, we lose orders of magnitude of money. Can you just keep it up and running? Can you make new ones? Whatever. Just do it. I don't care how cool it is or what the eye diagrams look like. I just need this thing to work."
Yeah, it's so weird. It's so weird going from designing something from net new and all the pressure that goes into creating that product. And then when you have to reverse-engineer something, there's usually some crunch or constrain on I need this data in order to do something new or keep it up and running.

Zach Peterson:

You mentioned that this kind of skillset is not something that's necessarily taught I guess in school or in training courses. And I think you're correct. I think if you just wrote down the list of skills needed to do this successfully, people know the skills individually but I think it's like the thread weaving them all together into a process that people don't really know unless they've tried and failed or tried it a few times with an old board and then really figured out a way to do it efficiently. Would you say that's fair?

Ethan Pierce:

I completely agree. And I recently did a training with one of the world renowned microsoldering trainers, Jessa Jones, down in Honeoye Falls. And I mean her background is as a PhD in microbiology and genetics. And the way she applied that thinking into troubleshooting and reverse-engineering high-density layer count boards and iPhones, that's insane.
I think it's not just writing them out like the skills, it's also taking that abstract thinking and stitching them into that thread that you described of a process. I know these things but I got to put all these things together to make a dataset of a thing that then I can manufacture again or make actionable design decisions on and production decisions on.

Zach Peterson:

You also brought up the whole set of data earlier that's really needed to make a product into an actual product. And I think when some designers maybe who are newer think, "Oh, I have the Gerbers so I can reverse-engineer this and recreate the layout." I mean, that doesn't capture what the purpose of the product is. You certainly don't capture anything like the firmware obviously. You certainly don't have any information on what the components are unless you can decode designators and kind of infer it. Maybe you have the BOM.
So, there's a lot more data that goes into this than just, "Hey, I've got some Gerbers, can I remake a layout?"

Ethan Pierce:

Oh, absolutely. I was just thinking through this, too. It's like just because you have a dataset, one, doesn't mean that it's right. And you should go through and understand, try to pull out as much of that story from those design files and those gerbers as you can because you might find that once you've gone through the exercise of building back those Gerber files into a design that you see that, gosh, that designer really made a mistake. And I would not have done it that way.
But you wouldn't know because I think it's kind of that story of the monkey that climbs up the tree and grabs the banana and the other monkey that grabs it and pulls the monkey down. And eventually, that monkey that reached for that banana isn't in the group anymore. And you assume that, "Oh, this is just the way that it is. This is a real dataset. This is the truth."
It's not the truth. It's just one person or one organization's attempt to build something based on a set of assumptions. And now, it's your job to not just take and copy paste it or copypasta, whatever you want to call it, into your new thing. It's like take what somebody did, bring it into a new product and go through the engineering design process.
I wanted to make a comment on this which is I think whether you're reverse-engineering or you're retrofitting a system, I don't know if in some people's minds, it's, "Oh, well, this thing is done. It's done. It exists. So, I should be able to take this thing and in a very short amount of time get something usable." Well, I think you and I were talking about this, it's not always the case because you have to go through the exercise to not just recreate it but you also have to understand what did somebody do? It's a brand new product cycle.
And when I tell people who want to retrofit a system versus creating a net new ecosystem of products, it's like you're going to have to treat it as a new thing and not as this, "Oh, well we're going to save money by reducing the time of development." If you're reverse-engineering something, there really has to be a solid reason why.
And Zach, you mentioned to this too when people approach you and they're like, "Hey, I've got a design to reverse-engineer." And even it's similar with this project that I'm doing currently for a customer. It's like, yeah, you've got this thing and, yeah, you have the Gerbers. And, yeah, you have a schematic. And hey, there's this black box thing that we may or may not be able to figure out.
Is it really worth it to go down the rabbit hole to reverse-engineer a CMOS gate array that was burned in 10, 15, 20 years ago when it makes sense to just take, if we got a board that works, just reverse-engineer, enough of the functionality and recreate a net new system versus just doing this interesting engineering exercise of, "Can I figure it out?" Of course you can. If you've got the thing, of course you can figure out what everything means.
Maybe there is some stuff that you can't but at some point, money will be able to solve all problems. I've got colleagues that run a silicon reverse-engineering team where they can decap something and they can do a block diagram and they get paid lots of money to do it. But a lot of the times, that's not time-efficient and it's just not what the customer needs. At the end of the day, it's where are we trying to go? Typically, it's keep a system online or it's move forward with a net new product and recovering something interesting about a past design.
 

Zach Peterson:

You brought up a really good point here and I was going to bring up the cost aspect. But I guess related to that is if you're reverse-engineering a 20-year-old system, let's say some of those parts might not even exist anymore. You brought up the CMOS gate array. Like today, no one would do that. They'd throw an FPGA on there or they would do firmware in C or something and they'd throw it on a microcontroller.
No one's using the CMOS gate array. The last time I had any experience with the CMOS gate array was 20 years ago when I was in college. And it was part of our electronics class. And even then, they were going out of style.
 

Ethan Pierce:

Yeah. And it took me a couple weeks to find this part because the original dataset, I had a photo of the board but not the right side of the board. I had a schematic that a bunch of people spent time to recreate. And they put the wrong vendor name. So, here I am going down this rabbit hole of a company that got acquired by Panasonic when that wasn't even the company.
I eventually find the data sheet and I'm like, "Oh my God, this was a thing that when you designed with this, it was a plugin into an ECAD tool and then you'd send that off to this magical land where it would get fabbed." And that just to me in today's day where you've got FPGAs or you've got microcontrollers that their IO is so fast, you can do much of that same implementation. It just blew my mind.
 

Zach Peterson:

Yeah. This comes back to the cost aspect because at that point, I think people might go down that rabbit hole of trying to reverse-engineer because it's perceived as a cost-saving measure.

Ethan Pierce:

Right.

Zach Peterson:

But the reality is that if you actually look at what you have to do and all of the steps involved to correctly reverse-engineer something and get all of that information about the product, you're probably going to spend just as much time and energy and money to just engineer it from scratch as a new revision.
 

Ethan Pierce:

Yeah, yeah. Now, I will say that we touched on this which is sometimes especially even in a super high-dense system, sometimes you might be able to find an embedded Linux system but there's some interesting IO. You'll probably be able to find a reference design most people are aware of the logic for, whatever chipset and it's got DDR ram and it's got some NAND flash on there.
But the IO side, whatever witchcraft is going on over here, you might be able to find a reference design or you might be able to get enough of the reverse-engineering done that you don't have to do the rest of the system. But I've definitely seen designs where they had some wizard engineer who for that timeframe implemented this design with a particular set of discreet components for cost-savings. And that's how they were able to accomplish the goal.
And I think thinking about, okay, well, if this design was done 20 years ago, pulling out that design and trying to figure out, okay, well what was the purpose? You mentioned that. What's the purpose of this thing? Can we do it with something that's already in a chip down thing where in the last 10 years everyone's been stuffing it into their microcontrollers? Of course. Why should we recreate this externally?
Or there's a chip that already exists that does the same function and you don't have to buy all these other components in order to just do the same thing, especially with the silicon shortage that continues to be ongoing. If you can reduce risk in your BAM, even if you're going down to an integrated part, from what I learned from my time at Apple is that if there is a part that's interesting in the industry, whatever it is, like wireless charging or something, if there is a large enough company that's designing something in, other companies are going to be interested and motivated to recreate that same silicon.
So, it's kind of trusting the industry and especially with APEMs, everyone has a, what is it, the 748?
 

Zach Peterson:

Something like that.

Ethan Pierce:

Everyone's got a spin of that.

Zach Peterson:

Yeah. So, you brought up something interesting here too which is the fact that sometimes a highly integrated part might exist for that function. So, this is an opportunity to possibly save yourself quite a bit of headache and probing in in a PCBA because if you know the purpose or you've kind of decoded the purpose of what the original product was supposed to do, you can identify, "Hey, this big block of 15 components just does this function and there's an ASIC for it or there's 10 ASICs for it.| Just go with the ASIC.
 

Ethan Pierce:

Yes. Yeah. And sometimes I was thinking about this where when you're reverse-engineering a design, humans are really good at dealing with different systems and bolting and plugging things together. But when you've got a design, it's hard to see the broader thing until you get it into a computer and paper.
And I think if you can understand the general function or the purpose of this 15 discrete component thing, if you learn enough about it where you're like, "Okay, I get that this is like this A to D converter and it does this wacky thing with this gate array or whatever with this thing and there's an output. I know this is a block. I'm going to go find this ASIC that does something similar so then I can just file this away in my dataset as, all right, this is black box. I know what the function is. I don't have to be concerned with it anymore."
And that's going to end up saving you time rather than like, "Whoa, how do these things go together?" Yeah, if you're up at 2:00 in the morning and you want something to think about, sure. But again, there's a purpose. There's typically money and time on the line. But you're absolutely right. If there's an ASIC, just stop making your life more difficult for yourself.
If somebody already put the time and effort into it like we're built on the ... All of this electronics industry is built on the shoulders of giants. So, save yourself some time and solve more interesting problems further down the line, because there's definitely going to be tougher ones that are more interesting than what do these 15 components do and how do they all twiddle together to make it do this thing?
 

Zach Peterson:

This is reminding me of a conversation I just had recently with a friend who works in the semiconductor industry. And you brought up standing on the shoulders of giants. He made the remark that it's almost like some of these systems that are out there and are operational in the world have been around so long that we don't even know how they work anymore. Or we have to go back and relearn all of that knowledge for how some of this stuff works.
It's gotten so deep. And there's been so much time between the time the builders were active and working on it and now the people who are in place overseeing it that that tribal knowledge is gone.
 

Ethan Pierce:

I just want to touch on this for a second. It's like you have industrial systems. So, there's this fascinating wax 3D printer from Thermojet. It's this 3D system Thermojet printer. I think it runs Windows NT on. It's like got this old board.
But here's the thing, the print quality is unbelievable, Zack. Because the way that they're doing is they've got a tectonics head and it does this inkjet spray and then they've got a planerizer. The problem with the product is they never assumed when they designed it that people would just go and take the wax parts, melt them down and put them back into the machine.
And so, now you've got this industrial system that is super, super high end that teams are just trying to keep up and running. But for that original business, they're not motivated to keep that product up and running. And sometimes they have to change the design for the business. So, sometimes you've got this mix of tribal knowledge and it's like, "Okay, I don't even remember how to interact with this industrial machine to make this thing work." And some things that are great technology fade out and other things that maybe are subpar or enough of an equivalent replace them.
But my philosophy is take as many pictures, take as many pictures, photos, notes. Just anytime you ... If you're working on something, just talk to the person who did something to it. Even if you're talking to a technician, you'd be surprised if you talked to the technician that worked on this system. Somewhere buried back in their closet, they've got this little poster roll and it ends up having these sun-faded schematics. And that might be the difference between you being up for the next six months and being able to actually have eight hours of sleep.

Zach Peterson:

Yeah. You've also brought up something important which is keep physical copies. I mean, everybody's turned into a digital packrat. I know I have. I feel like my laptop is constantly running low on memory because I am so low to hit that delete button sometimes on stuff that's five or 10 years old that I just drag around with me from computer to computer.
But there's something to be said for physical copies because maybe the original builder of the system, they don't have any of that stuff anymore at least digitally. Or maybe they have gone through some sort of merger or acquisition and a bunch of stuff got lost. And I mean that you never know what happens.
 

Ethan Pierce:

Absolutely. And there's this scanning system that it's a company based out of Santa Monica. They make this really nice laser scanning system. But the software that they have, they just happen to have this activation server that's still up and running. And you have to go find the machine. If you get a new computer, you have to go get your old machine up, run that software, pray that it's still working, hit that activation server, unlock it and reactivate it on your new machine.
But I think we get really excited about, "Oh, there's this new technology and there's this standard and there's this thing." And I think as I've continued to age in the community, what I've found is when I was really sold on some new Kodak or file format or whatever is the thing, "Oh, it's way better than this one." It's like can you make a bet?
And when you store that data, making sure that you're storing the accompanying software and some system to recover those digital files because so often, you will store something in a file format that you're like, "Oh yeah, I'm going to have that later. It's great. No problem." And then you go and you try to open it up and you can't run it. I mean, there was this whole thing with Apple's transition from Intel to Apple Silicon.
And there's a lot of software that just won't run, not because it's not going to run because of the X86 interpretation layer, there's some API that just won't translate and you can't open this software. So when you're storing digital data, making sure that you've got a set of software that you walk away with it in addition to a physical machine.
I used to think that my father was nuts for keeping around all these old SGI machines and old Dell. And they end up becoming so invaluable because just to be able to boot it up, open that CAD package from 2000 whatever, load up that native file and look at that data, dude, it becomes priceless.

Zach Peterson:

We've talked to customers and they have remarked that because they had an older CAD system before maybe they started using Altium, and they have all these old files that they periodically need to access because they're in an industry that has products with 20 and 30-year lifetimes. They keep around an old box, just like you're saying, with a perpetual license on that box just so that they can open that file and make sure that they can still access it. And if needed, they could actually visually go back and forth between a new version and the original version. And then they can do whatever they need to do on that product and maintain it.
 

Ethan Pierce:

Absolutely. And I'll tell you, I got a little bit cocky growing up in this being a '90s kid and growing up with technology. And I was like, "Oh, I'll just virtualize all these things." But there's so many industrial tools or CAD packages that need all these little dongles and things or drivers or cards or things that you just keep the machine, keep it up and running.
If anything, make a DD copy of that hard drive. Just find another similar size, IDE drive, whatever, and just keep a freaking copy of. So that drive ever goes tick, tick, tick, you can just throw it back in there.

Zach Peterson:

Oh, man. Yeah. I remember the tick, tick, tick on the magnetic drives.

Ethan Pierce:

Oh, man. Not fun.

Zach Peterson:

We have it so easy on consumer electronics. And I think it's easy for us to then assume that all of the industrial systems that we rely on and things like telecom and military and defense and all this stuff is also being constantly modernized. But it's not.

Ethan Pierce:

No.

Zach Peterson:

Not at all.

Ethan Pierce:

But I think that that comes from ... Go ahead.
 

Zach Peterson:

Especially in like MilAero because MilAero has had the attitude for so long that if it ain't broke, don't fix it.

Ethan Pierce:

Absolutely. And I think in order to ... When you approach a product that was designed in that space, taking a step back, I think it's important to recognize that when we have products that exist that are in the flesh that we interact with, whether they're personal consumer products or they're things for work, somebody spent the time to engineer and design that product using some set of assumptions, tools and processes.
And I think that we sometimes have a tendency to meld those things together and create these false realities for ourself that you just described of like, "Oh, well, of course in the MilAerospace, they're going to modernize things." Well, it's like, "Well, no, they can't because there's a set of approvals that they have to go through. It has to meet some technology readiness. It has to have the support contract from that vendor to commit that I'm going to be spinning this stuff for the next 10 or 20 years." And it's an interesting bias in the engineering space to be aware of.

Zach Peterson:

Yeah, I think I agree with you. And it's interesting that we do create those kind of false narratives or realities about how everyone else operates just because myself or my company operates a certain way, especially when it comes to modernization. And the MilAero modernization is always interesting because there's such a huge cost associated with that, too.
I mean, you're talking about modernization of these systems where it's like a million bucks a system per unit. And they may have 10,000 in the field let's say. I mean, how are you going to go through and modernize that? That's crazy.
 

Ethan Pierce:

Yeah. You can't. You have to go through and evaluate, well, where is that part of the business making money? Is it just a thing to sustain things? Are we losing money? And just like, how are we going to fix things even outside of the MilAerospace?
When I've been doing connected internet, internet of things devices. So, honestly, the average cost is between $200 and maybe $400. It hovers around for industrial connected devices around $300. So, you're like, "Okay, 300 bucks, that's cheaper than 2 million bucks or 5 million bucks for a board spin. And there's X number of them in the field."
Well, once you start doing the economies, the economies we scale for, you're like, "Okay, I've got this $400 board that, yeah, I can see it on my desk." But I had to go and I had to, from my business, pay a technician or somebody to drive out to the middle of nowhere and install it on a silo tower.
Or one of my favorites was an industrial monitoring system that goes inside the electrical vaults in the streets, in cities in order to monitor. As the insulation around the wire starts to fail, the conductors between the wires have the chance to spark. And because the insulators are these different polymer compounds, they break down and there's this gas and they are heavier than air.
Now, what becomes really interesting and you can Google this on YouTube, it's unbelievably wild, is the insulator degrades. And the conductors as they're degraded, eventually they spark against each other because you've got high voltage lines through there and you've got this gas. And what happens is you have a two, three, whatever the several hundred pound solid cast iron manhole cover go hundreds of feet in the air. So, this is a real problem.
Now, there's a company that makes a solution for it. But what was wild besides the fact that they said, "Hey, why can't we get cellular coverage with the antenna underneath this cast iron thing in this vault?" And I was like, "Listen, I don't know what to tell you. I can't change Mother Nature. We're just a startup."
But the fact that in some places, this $300, $400 device, you would have to get sometimes a quarter million dollars' worth of permits and time to shut a roadway down in the middle of Boston just to take a manhole cover on and install this $300 box. So, sometimes just like we're talking about, it's not economical. Just because the thing isn't expensive doesn't mean that the overall cost of replacing it is even feasible.
 

Zach Peterson:

Yeah. I didn't even think about red tape.

Ethan Pierce:

Oh, yeah. Well, it's hard for us to think about the red tape because we're so focused on creating the physical thing and the system. And then somebody else in the business team is like, "Oh yeah, three months ahead of time, I got to apply for this permit and shut the street down that's in the dead center of Boston."

Zach Peterson:

By the way, in that three-month time, the manhole cover might fly off the street anyways.

Ethan Pierce:

Oh yeah, yeah, absolutely. But yeah, you make a great point about how do you service and replace things. And I think as creators, as builders and designers in this field where we love to take ... We're taking raw materials and we're creating something. It's really this artistic outlet that we get.
It's important to recognize that there are times when we get to really express that. But there are other times where we have to step outside ourselves and work with our cross-functional teams and be able to ask questions, be the idiot in the room and learn more from the business people and from the operations people like, "How does this thing work? Tell me more. What do I need to know? What's your process? How does this get installed? What's the service? What's the end of life service? What's the warranty?"
Asking dumb questions like, "Well, what are we going to do in 10 years when all of us are retired and we're gone? How is this going to keep being maintained?" We signed this contract and this thing is going to be in the field for the next 20 years.
There is some of the motivation to say, "Look, it's somebody else's problem. It's future self's problem." But I think it's worth it to take a step back and ask questions. Because we've already got that curiosity on how to create, so why not use that curiosity to learn about how the rest of the business operates? Because it's really going to fascinate you once you start asking those questions.

Zach Peterson:

There's another piece to these devices that might need to be reverse-engineered sometimes, which we kind of mentioned tangentially but it's the code base. If they're running an embedded application, and imagine if that code base gets lost, but you have to do an update on that device, well then, what do you do?

Ethan Pierce:

Oh, man.

Zach Peterson:

I mean, if the firmware exists in memory somewhere, can you extract it? Do you have to go through JTAG and probe it and figure out what the logic is? How do you even approach that?

Ethan Pierce:

Gosh, I think there is something outside of the design space in this software-embedded world. You have to hope and pray that some of the things are there for you to be able to recover them. So hopefully, somebody put the memory or put the binary in EEPROM somewhere where you can dump it.
There's enough tools that exist that are open source where you can take something. And there is an entire art form where you can dump the binary and you can get the instruction set. I mean, that's way out of my knowledge base but it's there where you can take ... If I've got the binary, I can dump the instruction set, or not the instruction set, I can dump the firmware ins instructions of what's going on in the firmware. And then I can build back the functionality.
But that's still pretty tricky. The other thing that happens is you've got to design and somebody locked the memory. And you have to do something that Joe Grand would do and glitch the system in order to be able to get into it to dump that stuff.
I do see with organizations a lot of the time, when you're building industrial IOT devices, they're like, "Oh, well, we need to be able to disable JTAG." And I'm thinking to myself, and this was a philosophy from some of the other engineers that I've worked with, is assume that if somebody has physical access to the device. They're going to be able to get everything that they need out of it. I mean, you can do something like Apple where you've got an EEPROM where if you bork that EEPROM, the whole thing kind of zeroizes itself.
But if somebody has enough money in time, if they have a physical thing, they're going to be able to get the stuff out of it. But in terms of doing updates, recovering a firmware code base, it's this embedded challenge of, "All right, well, how do I get this binary out of this device?" And then going through a total another set of skillsets and tools of reverse-engineering the binary, getting what the different instructions are and kind of reverse-coding back what the functions are.
Hopefully your device is simple enough where you can use a logic analyzer. There's some digital protocol with I2C or SPI and you can just watch some bites fly by and build back up that protocol rather than actually trying to figure out what the heck somebody did in that device.
And I would also argue that you could go through the exercise of if you had that skillset to reverse that binary and build something back up. But I would argue that if you had a lost design and you had to recover the firmware and be able to provide a firmware update, it makes more sense to triage and build up the characteristics and performance of the system and then write some net new firmware. Then go down the rabbit hole to reconstruct this thing in bare metal just to provide this update.
Because you're going to have to do it one way or another is you're going to have to go through your testing. You're going to have to go through qualification. You're going to have to train your technician. You're going to have to build the support documentation. And it's that thing we referenced earlier in our conversation which is like, "All right, when do we cut off the reverse-engineering? When do we move forward and unblock this so that whoever this is for, they can continue to operate their business to save money, make money or ensure compliance?

Zach Peterson:

One thing that comes up sometimes on the internet if you're ever browsing forums is decompilers. So, if you can get the binary, can you decompile it back to something that allows you to more easily reconstruct source code? Or are you SOL and you're going to have to probe iOS to figure out logic in gives me this logic out?

Ethan Pierce:

That's the word that I was hunting around for my brain. The decompiler is going to give you what the machine code is of the binary. But it's not really going to give you this Rosetta Stone of, "Oh, well there's this class function, there's this." You still have to go through the effort to build it.
So now, while you're not SOL, outside of firmware engineering, it's this other way of thinking in how to not just ... Okay, so you learn this path of, "All right, I got a binary, I'll decompile it." But now, you have to build up this other muscle and skillset of, "Here's what it says. Now I got to go build back the firmware."
And honestly, if you can't find it in the embedded space, there is quite a plethora of people on YouTube that it's unbelievably fascinating. They will take malware viruses and they will decompile those. And they will then reconstruct how the thing actually works. So, if you can't find the skillset in this particular thing that you're trying to do which is firmware or whatever, somebody else might be trying to do the same process using effectively the same steps, just a different set of tool but flexing that same skill.

Zach Peterson:

Yeah. See, I think there's this perception that decompiling gives you the C code but it doesn't, because there are multiple ways to write C code or whatever other language that then compiles down to these machine instructions that eventually gets translated to binary for your particular chipset or your particular architecture. Is that the right way to think of it?

Ethan Pierce:

Yeah. Yes. I would think about it like that. And it's not just that it's C code that gets compiled into this instruction set, it's whatever that engineer was thinking when they wrote it.

Zach Peterson:

Right.

Ethan Pierce:

And as we move more and more further along in technology ... I am honestly seeing, Zach, I have seen a company deploy CircuitPython at scale which is unbelievably mind-blowing, where most embedded engineers are like, "No, no, no, no, no, no, no. You want to chip something, it's got to be straight C, bare metal. I'm not even into RTOSs."
But we're moving into a place where people are using interpreted language in embedded systems deployed in the field. It's mind-blowing. But it's important to remember, even if you have the binary, again just like a system where you've got the dataset, you've got to build it back into, "Well, what was this person thinking when they wrote it?"
And look, if somebody's interested in doing that, I think that they should flex that muscle. I don't know that there ... There are not enough people capable of doing good firmware reverse-engineering. And the people that are good at doing firmware reverse-engineering and who haven't smoked marijuana are in three-letter organizations. I mean, that's just how it is.

Zach Peterson:

That's what I was just wondering. The people who are doing that are probably not working in the commercial sector. They're like you said, working somewhere in government in the basement. They don't let them see the light of day unless they absolutely have to. And they're the ones that are responsible for taking apart and really decoding this stuff that we don't even know about as just regular citizens.

Ethan Pierce:

Absolutely. And even in the large companies where you do have people that are really talented, that individual doesn't get exposure to the rest of the industry because they're wicked smart and that company needs to keep them. I mean, there was a story where at a company, at a tech company in Canada that made phones, they had an engineer who was able to take their platform and get Objective-C running on their platform and able to emulate and run iOS applications.
This was years ago. No one's ever talked about this. This doesn't exist. But they basically just hid the whole project. So, even if there's somebody that's capable of doing that work, they're going to hide it. You have to hide it.

Zach Peterson:

Wow. That's interesting. Well, is that just because companies don't want people or other companies, I guess, poaching their top people?

Ethan Pierce:

I think it's this a little bit of security through obscurity. But also at the end of the day, money is still what drives much of this work. And if you're an engineer and you like doing these things, somebody's going to pay to do it and you're going to be happy. And you're not going to think through like, "Oh, well, this needs to be out in the world."
And there's just so many factors that go into somebody's decision of work that I am deeply appreciative when there are people that come into the community, like a Joe Grand who put all of this documentation out there to teach people how to actually reverse-engineer or do hardware hacking. It really is a gift to the world and I think allows humanity to move forward.
And while there are people that hide in the organizations, eventually, hopefully, they retire and they help people out do stuff. But yeah, it's a fascinating world that has a lot of desire. But I don't think enough people know how to get into it. It's definitely weird and tricky to even get into it and be productive in it too.
 

Zach Peterson:

I think the other place where you might see this and where someone might be much more open about what they're doing is in academia.

Ethan Pierce:

Absolutely.

Zach Peterson:

And then they publish some paper about how they reverse-engineered or hacked into some chip or some sort of security thing that they did. I mean, the list goes on and on. But they're going to be much more open about it because they have an incentive. They need to publish. It's going to move them ... It's going to bring them prestige. It's going to bring them research dollars, that kind of stuff.

Ethan Pierce:

Absolutely. And I have a colleague who's in the federal government in one of the three-letter organizations. And he goes to DEFCON and he keeps tabs on all these people doing different transportation things. And it's really interesting and important for him to stay abreast of those things but really have open dialogue.
And I think that most of these organizations in some capacity, they're not out there to get people. They do want to collaborate with those researchers and encourage the research because whether or not we like it, exploits and exploits exist. And they're at some point in different things, whether it's an iPhone or whether it's a transportation system. There's a dollar value on those exploits.
And to keep an open dialogue to exchange information I think reduces the risk of bad things happening to people. It's like, "Hey, there's this traffic system," or, "There's this phone thing. We need to fix this so people's photos or this traffic system operates properly." Having that open dialogue I think is really important. And their organizations are willing to do that.
Now, do I think bug bounties are high enough dollar value to incentivize people to disclose those? I think sometimes not enough, but anyways.

Zach Peterson:

Well, that's much more of a software thing-

Ethan Pierce:

Absolutely. Absolutely.

Zach Peterson:

... than it is a hardware thing.

Ethan Pierce:

Absolutely.

Zach Peterson:

I mean, if you're a company, what are you going to do? Everybody that wants to participate in a bug bounty, you send them out a board and send them a board in the mail and they start probing it with their DMM.

Ethan Pierce:

I mean, the wild thing is there is a security program that I saw came out from Apple where they would basically send an unlocked device for people to do that.

Zach Peterson:

Oh, really?

Ethan Pierce:

Yeah. That was wild.

Zach Peterson:

Wow, that's interesting. Okay. I had no idea that was something that Apple would be willing to do.

Ethan Pierce:

Yeah. I mean, you keep ... It's not just like anybody can go request it. It's again those academic researchers.

Zach Peterson:

So, you have to apply for it and you have to tell them who you are, where you work, that kind of thing.

Ethan Pierce:

Yeah. I mean, to bring it back to hardware, there's a quote that I had heard at one point which is software comes and goes but hardware is forever.

Zach Peterson:

That's what makes us different, I guess.

Ethan Pierce:

Yeah. And I wish that the ... The hardware industry, I find the creators, designers product people is so much more tight-knit because there's not many of us because software tends to eat the world. But you still have to have people that know how to design the systems correctly. You can't just keep copying and pasting these reference Linux systems and just shoving them out into the world. Somebody's got to know how to do it correctly. Yeah.

Zach Peterson:

This has all been extremely interesting and we're getting a little low on time. But I do want to make sure that we get the announcement for the webinar once again here in the podcast.
Ethan Pierce will be giving a webinar on all of this stuff on April 11th. It's through the Printed Circuit Engineering Association. And we will include a link in the video description as well as in the show notes. So, if anyone is interested in these topics, I would encourage you to go and attend that webinar. I will be trying to carve out some time in my schedule to attend it as well.
So Ethan, I want to thank you so much for joining us today. This has been extremely interesting. I love talking about this kind of stuff, especially with reverse-engineering because it's something that I periodically get asked about and sometimes I don't know the right answers. So, this is extremely interesting. Thank you.

Ethan Pierce:

Absolutely. Zach, thanks so much for having me. And at the end of the day, I think the most important part is just to stay curious.

Zach Peterson:

Oh, I totally agree. Yeah, never stop learning.
We've been talking with Ethan Pierce today, Director of Hardware Engineering for Pierce Design. If you're listening on YouTube or you're watching on YouTube, make sure to hit the subscribe button. You'll be able to keep up with all of our podcast episodes and tutorials as they come out.
And finally, don't forget to sign up for the webinar. It's going on April 11th. You'll find a link in the show notes as well as in the video description. And last but not least, don't stop learning. Stay on track. And we'll see you next time.