Re: Paper announcement: Is finding security holes a good idea?
On Wed, 21 Jan 2004, Eric Rescorla wrote:
> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
It is a very weakly justified assumption that the number of Black Hat
exploitations following Black Hat Disclosure is 'almost certainly less'
than the 'peak rate after (Public) disclosure'. The rest of your paper
rests heavily on that assumption - and it in fact nearly pre-determines
your conclusion.
It only takes _ONE_ more Warhol Worm from non-public security bugs than
publically disclosed ones to invalidate your argument (this is obviously
true in the _converse_ as well). This is because the _COST_ of such a worm
is magnified beyond reason by the network. Each individual worm represents
a large fraction of _all_ such intrusions and their cost. This is NOT a
large statistical universe. Individual events skew the whole dataset. One
major exploit may cost as much as thousands or even millions of smaller
exploits.
A bug allowing complete remote compromise of a box that is merely
_connected_ to the net is of a completely different magnitude than one
that causes Mozilla to mis-display a URL. Treating the two bugs as
comparable is a significant error. It is not justified to believe that
they are _generated_, _discovered_, _exploited_, or _fixed_ at the same
rates or that they have even order of magnitude similiar costs.
There remain too many large unknowns to justify _any_ good conclusions re
bug hunting and disclosure vs the cost of intrusions.
--
Benjamin Franz
On that of which one cannot speak, one must remain silent.
---Wittgenstein