The Hardware Hacker: Adventures in Making and Breaking Hardware

ACKNOWLEDGMENTS

Thanks to all the hard-working staff at No Starch Press for making this book happen. In particular, thanks to Bill Pollock for conceiving and sponsoring the effort, and thanks to Jennifer Griffith-Delgado for compiling, editing, and arranging my writing into the form of this book.

brief contents

preface

part 1

adventures in manufacturing

chapter 1. made in china

chapter 2. inside three very different factories

chapter 3. the factory floor

part 2

thinking differently: intellectual property in china

chapter 4. gongkai innovation

chapter 5. fake goods

part 3

what open hardware means to me

chapter 6. the story of chumby

chapter 7. novena: building my own laptop

chapter 8. chibitronics: creating circuit stickers

part 4

a hacker’s perspective

chapter 9. hardware hacking

chapter 10. biology and bioinformatics

chapter 11. selected interviews

epilogue

index

contents in detail

preface

part 1

adventures in manufacturing

1. made in china

The Ultimate Electronic Component Flea Market

The Next Technological Revolution

Touring Factories with Chumby

Scale in Shenzhen

Feeding the Factory

Dedication to Quality

Building Technology Without Using It

Skilled Workers

The Need for Craftspeople

Automation for Electronics Assembly

Precision, Injection Molding, and Patience

The Challenge of Quality

Closing Thoughts

2. inside three very different factories

Where Arduinos Are Born

Starting with a Sheet of Copper

Applying the PCB Pattern to the Copper

Etching the PCBs

Applying Soldermask and Silkscreen

Testing and Finishing the Boards

Where USB Memory Sticks Are Born

The Beginning of a USB Stick

Hand-Placing Chips on a PCB

Bonding the Chips to the PCB

A Close Look at the USB Stick Boards

A Tale of Two Zippers

A Fully Automated Process

A Semiautomated Process

The Irony of Scarcity and Demand

3. the factory floor

How to Make a Bill of Materials

A Simple BOM for a Bicycle Safety Light

Approved Manufacturers

Tolerance, Composition, and Voltage Specification

Electronic Component Form Factor

Extended Part Numbers

The Bicycle Safety Light BOM Revisited

Planning for and Coping with Change

Process Optimization: Design for Manufacturing

Why DFM?

Tolerances to Consider

Following DFM Helps Your Bottom Line

The Product Behind Your Product

Testing vs. Validation

Finding Balance in Industrial Design

The chumby One’s Trim and Finish

The Arduino Uno’s Silkscreen Art

My Design Process

Picking (and Maintaining) a Partner

Tips for Forming a Relationship with a Factory

Tips on Quotations

Miscellaneous Advice

Closing Thoughts

part 2

thinking differently: intellectual property in china

4. gongkai innovation

I Broke My Phone’s Screen, and It Was Awesome

Shanzhai as Entrepreneurs

Who Are the Shanzhai?

More Than Copycats

Community-Enforced IP Rules

The $12 Phone

Inside the $12 Phone

Introducing Gongkai

From Gongkai to Open Source

Engineers Have Rights, Too

Closing Thoughts

5. fake goods

Well-Executed Counterfeit Chips

Counterfeit Chips in US Military Hardware

Types of Counterfeit Parts

Fakes and US Military Designs

Anticounterfeit Measures

Fake MicroSD Cards

Visible Differences

Investigating the Cards

Were the MicroSD Cards Authentic?

Further Forensic Investigation

Gathering Data

Summarizing My Findings

Fake FPGAs

The White Screen Issue

Incorrect ID Codes

The Solution

Closing Thoughts

part 3

what open hardware means to me

6. the story of chumby

A Hacker-Friendly Platform

Evolving chumby

A More Hackable Device

Hardware with No Secrets

The End of Chumby, New Adventures

Why the Best Days of Open Hardware Are Yet to Come

Where We Came From: Open to Closed

Where We Are: Sit and Wait vs. Innovate

Where We’re Going: Heirloom Laptops

An Opportunity for Open Hardware

Closing Thoughts

7. novena: building my own laptop

Not a Laptop for the Faint of Heart

Designing the Early Novena

Under the Hood

The Enclosure

The Heirloom Laptop’s Custom Wood Composite

Growing Novenas

The Mechanical Engineering Details

Changes to the Finished Product

Case Construction and Injection-Molding Problems

Changes to the Front Bezel

DIY Speakers

The PVT2 Mainboard

A Breakout Board for Beginners

The Desktop Novena’s Power Pass-Through Board

Custom Battery Pack Problems

Choosing a Hard Drive

Finalizing Firmware

Building a Community

Closing Thoughts

8. chibitronics: creating circuit stickers

Crafting with Circuits

Developing a New Process

Visiting the Factory

Performing a Process Capability Test

Delivering on a Promise

Why On-Time Delivery Is Important

Lessons Learned

Not All Simple Requests Are Simple for Everyone

Never Skip a Check Plot

If a Component Can Be Placed Incorrectly, It Will Be

Some Concepts Don’t Translate into Chinese Well

Eliminate Single Points of Failure

Some Last-Minute Changes Are Worth It

Chinese New Year Impacts the Supply Chain

Shipping Is Expensive and Difficult

You’re Not Out of the Woods Until You Ship

Closing Thoughts

part 4

a hacker’s perspective

9. hardware hacking

Hacking the PIC18F1320

Decapping the IC

Taking a Closer Look

Erasing the Flash Memory

Erasing the Security Bits

Protecting the Other Data

Hacking SD Cards

How SD Cards Work

Reverse Engineering the Card’s Microcontroller

Potential Security Issues

A Resource for Hobbyists

Hacking HDCP-Secured Links to Allow Custom Overlays

Background and Context

How NeTV Worked

Hacking a Shanzhai Phone

The System Architecture

Reverse Engineering the Boot Structure

Building a Beachhead

Attaching a Debugger

Booting an OS

Building a New Toolchain

Fernvale Results

Closing Thoughts

10. biology and bioinformatics

Comparing H1N1 to a Computer Virus

DNA and RNA as Bits

Organisms Have Unique Access Ports

Hacking Swine Flu

Adaptable Influenza

A Silver Lining

Reverse Engineering Superbugs

The O104:H4 DNA Sequence

Reversing Tools for Biology

Answering Biological Questions with UNIX Shell Scripts

More Questions Than Answers

Mythbusting Personalized Genomics

Myth: Having Your Genome Read Is Like Hex-Dumping the ROM of Your Computer

Myth: We Know Which Mutations Predict Disease

Myth: The Reference Genome Is an Accurate Reference

Patching a Genome

CRISPRs in Bacteria

Determining Where to Cut a Gene

Implications for Engineering Humans

Hacking Evolution with Gene Drive

Closing Thoughts

11. selected interviews

Andrew bunnie Huang: Hardware Hacker (CSDN)

About Open Hardware and the Maker Movement

About Hardware Hackers

The Blueprint Talks to Andrew Huang

epilogue

index

preface

When Bill Pollock, founder of No Starch Press, first contacted me with the idea of publishing a compilation of my writings, I was skeptical. I didn’t think there would be enough material to fill a hundred pages. It seems I was wrong.

My mother often said, It doesn’t matter what’s in your head if you can’t tell people what’s in it, and when I was in seventh grade, she enrolled me in an after-school essay writing class. I hated the class at the time, but in retrospect, I’m thankful. Starting with my college application essays and up to this day, I’ve found the ability to organize my thoughts into prose invaluable.

Most of the material in this book was originally published on my blog, but as you’ll soon see, those posts weren’t puff pieces written to drive ad revenue. One reason I write is to solidify my own understanding of complicated subjects. It’s easy to believe you understand a topic until you try to explain it to someone else in a rigorous fashion. Writing is how I distill my intuition into structured knowledge; I only write when I find something interesting to write about, and then I post it with a CC BY-SA license to encourage others to share it.

This book includes a selection of my writings on manufacturing, intellectual property (with a focus on comparing Western versus Chinese perspectives), open hardware, reverse engineering, and biology and bioinformatics. The good editors at No Starch Press also curated a couple of interviews I’ve done in the past that were particularly informational or insightful. The common thread throughout these diverse topics is hardware: how it’s made, the legal frameworks around it, and how it’s unmade. And yes, biological systems are hardware.

I’ve always gravitated toward hardware because while I’m not particularly gifted when it comes to abstract thought (hence the need to write to organize my thoughts), I am pretty good with my hands. I have a much better chance of understanding things that I can see with my own two eyes.

My entire understanding of the world has always been built on a series of simple, physical experiences, starting from when I stacked blocks and knocked them over as a child. This book shares some of my more recent experiences. I hope that by reading them, you will gain a deeper understanding of the world of hardware, without having to spend decades stacking blocks and knocking them over.

Happy hacking,

—b.

Part 1

adventures in manufacturing

I first set foot in China in November 2006. I had no idea what I was walking into. When I told my mother I was going to visit Shenzhen, she exclaimed, Why are you going there? It’s just a fishing village! She wasn’t wrong: Shenzhen was just a town of 300,000 back in 1980, but it had exploded into a megacity of 10 million in less than 30 years. Between my first visit and the time I wrote this book, Shenzhen gained an estimated 4 million people—more than the population of Los Angeles.

In a way, my understanding of manufacturing over the years has mirrored Shenzhen’s growth. Before going to China, I had never mass-produced anything. I didn’t know anything about supply chains. I had no idea what operations and logistics meant. To me, it sounded like something out of a math or programming textbook.

Still, Steve Tomlin, my boss at the time, charged me with figuring out how to build a supply chain suitable for our hardware startup, Chumby. Sending a novice into China was a big risk, but my lack of preconceived notions was more of an asset than a liability. Back then, venture capitalists shunned hardware, and China was only for established companies looking to build hundreds of thousands of units of a given product. My first set of tours in China certainly supported that notion, as I primarily toured mega-factories serving the Fortune 500.

Chumby was lucky to be taken under the wing of PCH International as its first startup customer. At PCH, I was mentored by some of the finest engineers and supply chain specialists. I was also fortunate to be allowed to share my experiences on my blog, as Chumby was one of the world’s first open hardware startups.

Although meeting the minimum order volumes of our conventional manufacturing partners was a constant struggle, I kept noticing small things that didn’t square with conventional wisdom. Somehow, local Chinese companies were able to remix technology into boutique products. The so-called shanzhai integrated cell phones into all kinds of whimsical forms, from cigarette lighters to ornamental golden Buddha statuettes (more on this in Chapter 4). The niche nature of these products meant they had to be economical to produce in smaller volumes. I also noticed that somehow factories were able to rapidly produce bespoke adapter circuits and testing apparatuses of surprisingly high quality in single-unit volumes. I felt there was more to the ecosystem—a story that was being told over and over again—but few had the time to listen, and those who did heard only the parts they wanted to hear.

The financial crisis of 2008 changed everything. The consumer electronics market was crushed, and factories that were once too busy printing money were now swimming in excess capacity. I made friends at several medium-sized factories in the area. I started to inquire about how, exactly, these factories were able to so nimbly produce their internal test equipment, and how shanzhai were able to prototype and build such bespoke phones.

The bosses and engineers were initially reticent, not because they wanted to hide potential competitive advantages from me, but because they were ashamed of their practices. Foreign clients were full of corporate process, documentation, and quality procedures, but they also paid dearly for such overhead. Local companies were much more informal and pragmatic. So what if a bin is labeled scrap? If the bits inside are suitable for a job, then use them!

I wanted in. As an engineer, tinkerer, and hacker, I cared a lot about the cost to produce a few units, and a couple of minor assembly defects was nothing compared to the design issues I had to debug. I eventually managed to coax a factory into letting me build a part using its low-quality but ultra-cheap assembly process.

The trick was to guarantee that I would pay for all the product, including defective units. Most customers refuse to pay for imperfect goods, forcing the factory to eat the cost of any part that isn’t exactly to specification. Thus, factories strongly dissuade customers from using cheaper but low-quality processes.

Of course, my promise to pay for defective product meant there was no incentive for the factory to do a good job. It could have, in theory, just handed me a box of scrap parts and I’d still have had to pay for it. But in reality, nobody had such ill intentions; as long as everyone simply tried their best, they got it right about 80 percent of the time. Since small-volume production costs are dominated by setup and assembly, my bottom line was still better despite throwing away 20 percent of my parts, and I got parts in just a couple of days instead of a couple of weeks.

Having options to trade cost, schedule, and quality against each other changes everything. I’ve made it a point to discover more alternative production methods and continue shortening the path between ideas and products, with ever more options along the cost-schedule-quality spectrum.

After Chumby, I decided to remain unemployed, partly to give myself time for discovery. For example, every January, instead of going to the frenzied Consumer Electronics Show (CES) in Las Vegas, I rented a cheap apartment in Shenzhen and engaged in the monastic study of manufacturing; for the price of one night in Las Vegas, I lived in Shenzhen for a month. I deliberately picked neighborhoods with no English speakers and forced myself to learn the language and customs to survive. (Although I’m ethnically Chinese, my parents prioritized accent-free fluency in English over learning Chinese.) I wandered the streets at night and observed the back alleys, trying to make sense of all the strange and wonderful things I saw going on during the daytime. Business continues in Shenzhen until the wee hours of the morning, but at a much slower pace. At night, I could make out lone agents acting out their interests and intentions.

If there’s one thing those studies taught me, it’s that I have a lot more to learn. The Pearl River Delta ecosystem is incomprehensibly vast. As with the Grand Canyon, simply hiking one trail from rim to base doesn’t mean you’ve seen it all. I have, however, picked up enough knowledge to build a custom laptop and to develop a new process for peel-and-stick electronic circuits.

In this part of the book, you’ll follow my journey as I learned the Shenzhen ecosystem over the years, via a remix of blog posts that I wrote along the way. Some of the essays are reflections on particular aspects of Chinese culture; others are case studies of specific manufacturing practices. I conclude with a chapter called The Factory Floor, a set of summary recommendations for anyone considering outsourced manufacturing. If you’re in a hurry, you can skip all the background and go directly there.

However, hindsight is 20/20. Once you’ve walked a path, it’s easy to point out the shortcuts and hazards along the way; it’s even easier to forget all of the wrong turns and bad assumptions. There’s no one-size-fits-all method for approaching China, and my hope is that by reading these stories, you can come to your own (perhaps different) conclusions that better serve your unique needs.

1. made in china

Before my first visit to China, I was convinced that Akihabara in Tokyo was the go-to place for the latest electronics, knickknacks, and components. That changed in January 2007, when I first set eyes on the SEG Electronics Market in Shenzhen. SEG is eight floors of all the components a hardware addict could ever want, and only later did I learn that it’s just the tip of the Hua Qiang electronics district iceberg.

As the lead hardware engineer at Chumby at the time, I was in China with then-CEO Steve Tomlin to figure out how to make chumbys (an open source, Wi-Fi-enabled content delivery device) cheaply and on time. With prices like those at SEG, we were definitely in the right country to make at least the first part of that mission a success.

Shenzhen’s SEG Electronics Market, the new electronics mecca. Akihabara, eat your heart out!

THE ULTIMATE ELECTRONIC COMPONENT FLEA MARKET

When I first stepped into the SEG building, I was assaulted by a whirlwind of electronic components: tapes and reels of resistors and capacitors, ICs of every type, inductors, relays, pogo pin test points, voltmeters, and trays of memory chips. As a total newcomer to manufacturing in volume, I was blown away by everything I saw at SEG.

All of those parts were crammed into tiny six-by-three-foot booths, each with a storekeeper poking away at a laptop. Some storekeepers played Go, and some counted parts. Some booths were true mom-and-pop shops, with mothers tending to babies and kids playing in the aisles.

A couple of family-run component shops

Other booths were professional setups with uniformed staff, and these worked like a bar—complete with stools—for electronic components.

A swanky professional parts seller

No one at SEG says, Oh, you can get 10 of these LEDs or a couple of these relays, like you might hear in Akihabara. No, no. These booths specialize, and if you see a component you like, you can usually buy several tubes, trays, or reels of it; you can get enough to go into production the next day.

Looking around the market, I saw a woman sorting stacks of 1GB mini-SD cards like poker chips. A man was putting sticks of 1GB Kingston memory into retail packages, and next to him, a girl was counting resistors.

The bottom-left corner of this display was packed with all kinds of SD cards.

Another booth had stacks of power supplies, varistors, batteries, and ROM programmers, and yet another had chips of every variety: Atmel, Intel, Broadcom, Samsung, Yamaha, Sony, AMD, Fujitsu, and more. Some chips were clearly ripped out of used equipment and remarked, some of them in brand-new laser-marked OEM packaging.

The sheer quantity of chips for sale at a single booth at SEG was incredible.

I saw chips that I could never buy in the United States, reels of rare ceramic capacitors that I could only dream about at night. My senses tingled; my head spun. I couldn’t suppress a smile of anticipation as I walked around the next corner to see shops stacked floor to ceiling with probably 100 million resistors and capacitors.

Reels and reels of components, in every shop window

Sony CCD and CMOS camera elements! I couldn’t buy those in the United States if I pulled teeth out of the sales reps. (Some sellers even have the datasheets behind the counter; always ask.) Next, I spotted a stack of Micrel regulator chips, followed by a Blackfin DSP chip for sale. Nearby, a lady counted 256Mb DRAM chips—trays of 108 components, stacked 20 high, in perhaps 10 rows.

The equivalent of Digi-Key’s entire stock of DRAM chips sat right in front of me!

And across from her were a half-dozen more little shops packed with chips just like hers. At one shop, a man stood proudly over a tray of 4Gb NAND flash chips. All of this was available for a little haggling, a bit of cash, and a hasty good-bye.

A close look at a tray of 4GB flash chips

And that’s just the first two floors of SEG. There are six more floors of computer components, systems, laptops, motherboards, digital cameras, security cameras, thumb drives, mice, video cameras, high-end graphics cards, flat-panel displays, shredders, lamps, projectors—you name it. On weekends, booth babes dressed in outrageous Acer-branded glittery bodysuits loiter around, trying to pull you in to buy their wares. This market has all the energy of a year-round CES meets Computex, except instead of just showing off the latest technology, the point is getting you into these booths to buy that hardware. Trade shows always feel like a bit of a strip tease, with your breath making ghostly rings on the glass as you hover over the unobtainable wares underneath.

But SEG is no strip tease. It’s the orgy of consumer and industrial electronic purchasing, where you can get your grubby paws on every piece of equipment for enough kuai* out of your wallet. Between the smell, the bustle, and the hustle, SEG is the ultimate electronic component flea market. It’s as if DigiKey went mad and let monkeys into its Minnesota warehouse, and the resulting chaos spilled into a flea market in China.

Of course, a lot of the parts I marveled at in 2007 are antiques now. For example, 4Gb flash chips are trash, and 1GB flash disks are old news. At the time, however, those things were a big deal, and SEG is still the best place to get the latest tech in bulk.

THE NEXT TECHNOLOGICAL REVOLUTION

Three blocks down the street from SEG lay the Shenzhen Bookstore.† The first and most visible rack was a foreign book section, packed with classics like Stanford University professor Thomas Lee’s The Design of CMOS Radio-Frequency Integrated Circuits and several titles by UCLA professor Behzad Razavi. I picked up Lee’s book, and it cost 68 kuai, or $8.50 USD. Holy cow! Jin Au Kong’s book on Maxwell’s equations? $5. Jin Au Kong taught me Maxwell’s equations at MIT.

I went on a spree, packing my bag with six or seven titles, probably around $700 worth of books if I’d bought them in the United States. At the checkout counter, I bought them for less than $35, complete with the supplemental CDs, saving about $665. That’s equivalent to buying an economy-class ticket to Hong Kong!

In China, knowledge is cheap. Components are cheap. The knowledge in the books at the Shenzhen Bookstore was the Real Deal, the parts to use that knowledge are down the street at SEG, and within an hour’s drive north are probably 200 factories that can take any electronics idea and pump it out by the literal boatload. These are no backward factories, either. With my own eyes, I saw name-brand, 1,550-nanometer, single-mode, long-haul, fiber-optic transceivers being built and tested there. Shenzhen is fertile ground, and you need to see it to understand it.

Shenzhen has the pregnant feel of the swapfests in Silicon Valley back in the ’80s, when all the big companies were just being founded and starting up, except magnified by 25 years of progress in Moore’s law and the speed of information flow via the internet. In this city of 12 million people, most are involved in tech or manufacturing, many are learning English, and all of them are willing to work hard.

There has to be a Jobs and Wozniak there somewhere, quietly building the next revolution. But I’m a part of Shenzhen, too, and I still tremble in my boots with terror and excitement at the thought of being part of that revolution. This is my story, starting with that eye-opening trip to Shenzhen for Chumby.

TOURING FACTORIES WITH CHUMBY

In September 2006, Chumby was just a team of about a half-dozen people, and we had just given away about 200 early prototype chumby devices at FOO Camp, a conference put on by Tim O’Reilly. The devices were well received by the FOO Camp attendees, so I got the go-ahead to build the Asian supply chain.

Steve and I went to China to visit potential factories in November, but before we left, we had a trusted vendor in the United States give their best price for the job as a baseline for negotiations with the Chinese manufacturers. Then, we called up a lot of friends with experience in China and lined up about six factory tours. We hit quite a variety of places, from specialty factories as small as 500 people to mega-factories with over 40,000 people.

There’s no substitute for going to China to tour a factory. Pictures can only tell the story framed by the photographer, and you can’t get a sense of a facility’s scale and quality without seeing it firsthand. In general, factories welcome you to take a tour, and I wouldn’t work with one that didn’t allow me to visit. However, most factories do appreciate a week’s notice, although as your relationship with them progresses, things should become more open and transparent.

Speaking of openness, Chumby’s open source nature helped the factory selection process a lot. First, we had no fears about people stealing our design (we were giving it away already), so we’d eliminated the friction of NDAs (non-disclosure agreements) when sharing critical information like the bill of materials. I think this gave us a better reception with factories in China; they seemed more willing to open up to us because we were willing to open up to them. Second, there was no question in any factory’s mind that this was a competitive situation. Anybody could and would quote and bid on our job (in fact, we received a few unsolicited quotations that were quite competitive), so it saved a round of huffing and puffing.

After reviewing several manufacturing options, Steve and I eventually decided to work with a company called PCH China Solutions. PCH itself owns only a few facilities, but it has a comprehensive network of trusted and validated vendors, primarily in China but also in Europe and the United States. Not surprisingly, the factories that PCH subcontracts to were some of the best facilities we visited in China. PCH is actually headquartered out of Ireland—thus most of their staff engineers are Irish—so there was also no language barrier for us. (PCH engineers are also hardworking, resourceful, and well trained—and, as a bonus, they always seem to know the best place to find a pint, no matter where they are. I had no idea China had so many Guinness taps!)

There’s a lot to take in when you tour even one factory, let alone a half-dozen, and it’s easy to get overwhelmed and lost in the vagaries of electronics manufacturing. But there were some key details I found most fascinating during my factory tours for Chumby and in working with PCH to bring the chumby to life.

Scale in Shenzhen

One stunning thing about working in China is the sheer scale of the place. I haven’t been to an auto plant in Michigan or to the Boeing plant in Seattle, but I get the sense that Shenzhen gives both a run for their money in terms of scale. In 2007, Shenzhen had 9 million people.

To give you an idea of the scale of a Shenzhen factory, the New Balance factory there employed 40,000 people and had the capacity to produce over a million shoes a month. I estimate that from raw fabric to finished shoe, the process took about 50 minutes, and every perfectly stitched bundle of plastic and leather was sewn by hand on an industrial sewing machine. The stations are designed so that each stage in the process takes a worker about 30 seconds.

Of course, the New Balance factory is dwarfed by Foxconn, the factory where iPods and iPhones are made.

You know you’re big when you have your own exit off the freeway.

Foxconn is a huge facility, apparently with over 250,000 employees, and it has its own special free trade status. The entire facility is walled off, and I’ve heard you need to show your passport and clear customs to get into the facility. That’s just short of the nuclear-powered robotic dogs from the nation-corporation franchulates of Neal Stephenson’s Snow Crash.

Feeding the Factory

There’s an old Chinese saying: min yi shi wei tian. A literal translation would be people consider food divine or for people, food is next to heaven. You can also look at it as a piece of governing advice: the government’s mandate [synonymous with heaven] is only as robust as the food on people’s plates. Or, you can interpret it as an excuse to procrastinate: "let’s eat first [since it is as important as