Even though there have been far larger and more recent
incidents, this version hasn’t been updated as I’m preparing a
full website dedicated to this topic. It will have several 10-15 minute
videos broken into sections covering all content from this talk as well
as updates. Check eliogrieco.com
Who am I?
20+ years of professional programming
EDA (Electronic Design Automation) Software
Business Analytics as an Appliance
Seller Central Platform team at Amazon
What is a bug?
Undefined or Improperly Defined Behavior
Bugs are a break between the specification or intended behavior of a
program and the actual behavior of the program.
Programming is Deterministic
Bugs are always the programmer’s fault.
That said, programming is hard. Telling the computer
exactly what needs to be done in a way that always works is
Categories of Lethal Bugs
The program doesn’t do what is intended or expected.
Crashes, Hangs, and Delays
The program “works” but pressing the “obvious” button causes an
action to occur that is different than expected.
Interface is confusing e.g. Norman Doors
Scripting language is unclear
The program functions “as intended”, but interacts with psychology,
sociology and social norms in ways that are harmful or unintended.
Facebook and other social media
Google and other ad networks
Slack and other distraction-ware
Failure Points and Propagation
Good systems should tolerate subsystem failures without a total
Single Points of Failure
Not a Bug
This isn’t a talk about adversarial software issues. We won’t be
covering things that are intentionally harmful.
Hacking and Cyber Warfare
Killed 5 people
Designed by one engineer
No hardware interlocks
Patriot Missile System
Clock skew bug
Must restart every 24 hours
Air France 447
Stall indicator came on and then went off
No haptic feedback between control sticks
Boeing 737 Max 8
Bad aerodynamics, software used to “compensate”
No way to disable the automated system
Lack of redundancy in sensors
Lack of testing
Deaths due to sociological interactions
Just because you are not building “critical systems” doesn’t mean
your work isn’t capable of killing
There are at least a few more that are not known to the public. How
many exactly, we don’t know.
Why does this matter?
Software is Eating the World
Software will be in more critical systems
Systems that you don’t think are “critical” will be
Bad interactions will prove increasingly dangerous
Depending on undependable software will cost lives via lack of
Bigger, Faster, More Fragile
Reduced “friction” in and between systems will provide less
“dampening” and lead to higher volatility
Damping, friction and firebreaks are a good thing
The more you connect, and the faster you connect it, the faster and
further failures propagate
Opaque and Unpredictable
Trendy technologies are a plague.
AI is opaque and some algorithms suffer catastrophic
AI can be tricked
Tech Fatigue and the Techlash
People use what they trust
If software continues to be unreliable, people will likely stop
The economics will fail, breaches
are expensive and increasingly common
Solutions for the Public
Prefer Simpler Solutions
Passive solutions beat active solutions: Tricycle > Segway
Static websites work best for most content that isn’t a simulation
Prefer local compute to cloud
Prefer Open Source
It’s auditable: With enough eyes, all bugs are shallow
It can be maintained if the company goes under
Systems are made of other systems. Unmaintained subsystems make the
overall system less reliable.
Add Damping and Firebreaks
Damping should slow the spread and impact of an issue
Firebreaks should contain a failure or malicious action to one