500 Assembled Boards and All of Them Unusable

Recently, I received 500 assembled boards and all of them were unusable. It’s always one of the most exciting yet nerve-wracking times. You’ve been working on a new design, you’ve exhaustively tested prototypes and then you go to volume production. While it’s the same design as the known-good-tested prototype, it’s not necessarily the same “assembly instance,” if you will, so things can happen.

Are all of my BOM line items correct? Did I specify all the proper part sources, values and tolerances? I viewed my generated Gerber in Altium Designer’s Camtastic for that warm and fuzzy feeling of knowing the Gerber is correct. I mean after all, I saw the correct version of the board indicated properly on the silk screen layer. We used large enough SMT packages that, while tedious, enabled us to build our own prototypes. The new design was based on an existing design with the addition of a SPI bus. The SPI bus required pullup resistors. No big deal, just three 10K resistors.

To add the resistors to the design, I simply went into my Altium Designer schematic and shift-clicked-copied a 100 Ohm resistor symbol already in the design. After this, I updated the reference designator, resistance value and description parameters of the newly inserted resistor. I then copied that component two more times for my three resistors and wired them up to the SPI bus in the schematic. Finally, I imported the new parts into the PCB design file, routed the traces, performed DRC and was good to go.

I was a relatively unsophisticated Altium user at the time and used separate SchLib and PcbLib files for my library. Additionally, I always liked to provide the assembly house with at least an in-stock Mouser part number and sometimes other supplier part numbers. I put these in each schematic symbol as a parameter and included them in the BOM. As part of a small business, I had a variety of other tasks to handle (the majority of which had nothing to do with electronics design) and didn’t want various interrupts from the assembly house adding complexity to my day. I would also provide top and bottom photographs of the prototypes to clarify what side the manually inserted through-hole components were to be installed. Later, I associated 3D component views for all of the parts in our designs so that we could use Altium Designer’s excellent 3D capabilities.

To date, all our manufacturing runs had been right the first time. I’ve had years of design release to manufacturing experience and all of this was very familiar to me. This time, with the addition of only the three resistors, and one new IC, it was a simple change, and certainly the boards would function just as desired. I was fortunate to choose an assembly house that was located only a 45-minute drive away from our office and when I received the email that the boards were ready, I jumped into the car to pick them up. We were in the process of introducing a new product, and we already had advance orders.

After I arrived back at the office with the boards, I randomly picked out about five of them to check the quality of assembly and to make sure that the ICs were placed properly. I was always relieved to see that everything was in order. My next step was to power the board up for the essential “smoke test.” There was no smoke and all supply voltage measurements were just as expected.

The final step was to connect to the board and flash the processor with the test firmware. All progressed as expected. Surely this powerup test, firmware-flashing and check-out phase was just a mere formality at this stage of the game. The five prototypes that we built were all proven and nothing had changed. These boards were ready to go to final assembly and turned into real products. We had a backlog of orders and we needed to move forward and ship our products. But, I would still go through my acceptance test, assured it was a formality as was the case many times before. One of the firmware tests is intended to put the board in its lowest power state. Current draw should be in the microamps. But, as the meter reached the end of its range, I saw that current draw was considerably higher.

I tested additional boards and all had high current draw and were therefore faulty and unusable.  I was dumbfounded. Now the adrenaline was flowing, and it was time to find out why the considerable investment in boards resulted in 500 bricks. I needed to find out where the assembler screwed up, as I was sure it wasn’t something I did. It could have been poor solder joints, shorts in the bare board, or part reels that were not loaded into the pick and place machinery correctly. Was the orientation of a diode or polarized capacitor flipped? Perhaps a component location was left unpopulated for some reason? Maybe the boards were defective and there were severed traces due to over-etching? Was a key component defective? Was my Fluke meter defective?

As you have probably guessed by now, there was nothing wrong with any aspect of fabrication or assembly. Everything was done exactly how the data files directed. The problem was that those 10K pullup resistors that were added to the SPI bus were actually populated with 100 ohm resistors. This completely altered the supply current of this low-power, battery operated design.

Actual photo of the board as received with the incorrect resistor value populated.

The reason for this outcome was quite straightforward and quite frankly embarrassing. When I copied the 100 ohm resistor in my schematic, I did change the designator, value and the description of the part, but I did not change the recommended Mouser supplier part number. Here’s a screenshot of the actual BOM with the offending resistor part number highlighted.

As you can see the BOM does describe the resistor properly. It’s just that the part number provided was not correct. While this situation was squarely my fault, I did make the case with the assembler that the description on the BOM was accurate and that they should have seen the inconsistency with the supplier part number. Good assembly houses will usually double check the BOM and point out inconsistencies, but this one was not flagged. They agreed to re-work half of the boards and it was our responsibility to rework the other half. We were able to use the boards, but it came at a considerable cost in time.

Replacing 750 SMT resistors was quite tedious and served as a real lesson to me. Going forward, we put in more rigorous review processes and did not have a repeat of this situation. As we made these improvements, we concluded, that because our small team’s time was spread across so many non-technical domains, that it was actually more valuable to us to have a bit more formality in our design process.

If all of those boards were non-recoverable and destined for the scrap pile, our small company would have felt that impact in far greater ways than our higher volume, larger competition. Well-funded, larger companies can weather larger storms. In a smaller company, production runs can represent a larger percent of the company’s working capital. In my opinion, the formality of design processes, therefore, has proportionally more value to a smaller company, but as was the case with us, it is usually pushed back until something negative happens. Smaller organizations have an opportunity to set a foundation for excellence by using today’s compelling priced and easy to implement technology. Concord Pro, Altium’s on-premises component management product, is an example of such an opportunity.

If I was using Concord Pro, I could look for manufacturer parts first, just as I do when selecting parts at the beginning of the project. Once I found the 10K resistor I wanted to use, I could have acquired the component (including the symbol, footprint models, parametric data and more) directly from the manufacturer’s data into my Concord Pro managed library. I could even quickly add additional manufacturer part choices for the resistor to insure against stock or life cycle issues. Concord Pro’s templating capability brings additional rigor to the process and assures that the new components I create follow a standard “style guide” for uniformity in the information. I would build a new component, not merely a symbol linked to a footprint. Here’s an example of a component I created in Concord Pro:

I added two additional part choices in addition to the KOA Speer resistor I used. In the process, the supply-chain integration of Concord Pro enabled me to identify lower cost alternatives from additional suppliers.

I only need to review the component data once at creation. Then, I can use this component wherever it is needed. As I update the component with additional part choices, a revision log will be kept so that I know the history. If I had placed a “Component” rather than just merely a parametrically annotated symbol in my design, the odds of correctness would have improved exponentially. Rather than updating my design with a symbol that had parameter errors, I would have placed a known good verified component which contained all component related aspects in a signal placement from a component library that is my organization’s single source of component truth.

Even with the small size of our production volume, the delay in fulfilling orders, the delay in getting boards to final-assemblers on staff and the personal frustration of coping with this situation, would have more than covered the costs of placing Concord Pro into service and embarking on a more comprehensive component data management path.

Actual photo of one of the boards from the next manufacturing run. The 10K resistors are populated as designed.

The good news is that we found the problem with our boards long before the product made it into our customer’s hands. This is the value of overlapping validation. When your product is the “new kid on the block”, it’s also imperative to exceed your competition in terms of feature set, quality, and cost. Yet another consideration when deciding to invest in a more formal electronics design infrastructure. Altium Concord Pro is such an opportunity. You can learn more about it here.