[Haskell-cafe] Re: A question about "monad laws"
Richard A. O'Keefe
ok at cs.otago.ac.nz
Wed Feb 13 22:36:49 EST 2008
On 14 Feb 2008, at 2:28 pm, Roman Leshchinskiy wrote:
> Richard A. O'Keefe wrote:
>> On 12 Feb 2008, at 5:14 pm, jerzy.karczmarczuk at info.unicaen.fr wrote:
>>> Would you say that *no* typical floating-point software is reliable?
>> With lots of hedging and clutching of protective amulets around the
>> word "reliable", of course not. What I *am* saying is that
>> (a) it's exceptionally HARD to make reliable because although the
>> are well defined and arguably reasonable they do NOT obey the
>> laws that
>> school and university mathematics teach us to expect them to obey
> Ints do not obey those laws, either.
They obey a heck of a lot more of them.
Any combination of Ints using (+), (-), (*), and negate
is going to be congruent to the mathematically correct answer modulo
for some n. In particular, (+) is associative for Ints.
> It is not exceptionally hard to write reliable software using ints.
I did my BSc and MSc computing on a B6700, where the hardware *always*
notified you in case of an integer overflow. In that case, it was
perfectly easy to write reliable software. You just pretended that
the type 'INTEGER' in your program meant 'mathematical integer', and if
that got you into trouble, the machine was certain to tell you about it.
Using languages that do not check for integer overflow, even on hardware
(like, as it happens, both the different machines on my desk) that makes
it cheap to do so, I *have* had trouble with multiplying two positive
and getting a negative rules and also with a program that went into
loop because it happened to multiply two positive numbers and get
positive number that was smaller than the one it started with.
the problem dividing two negative integers can give you a negative
And one problem I definitely ran into was a Pascal 'for' loop with
bounds that ran forever.
When I contrast the amount of manual checking I have to do when
(or, for that matter, Haskell) with the amount of manual checking I
do when using Smalltalk or SETL or Lisp, and when I remember how life
*better* for me in this respect back in the 70s, well, it doesn't
This would be my top priority request for Haskell':
require that the default Int type check for overflow on all
operations where overflow is possible,
provide Int32, Int64 for people who actually *want* wraparound.
I've been told that there was a day when there was serious trouble in
US financial markets because the volume of trade exceeded the 32-bit
integer limit, and many programs started giving nonsense results.
Smalltalk programs just kept powering on...
> You just have to check for exceptional conditions.
Why should it be *MY* job to check for exceptional conditions?
That's the *MACHINE*'s job. When you bought your computer, you paid
for hardware that will do this job for you!
> That's also the case for floating point.
If you think that, you do not understand floating point.
x+(y+z) == (x+y)+z fails even though there is nothing exceptional about
any of the operands or any of the results.
I have known a *commercial* program blithely invert a singular matrix
because of this kind of thing, on hardware where every kind of
exception was reported. There were no "exceptional conditions", the
answer was just 100% wrong.
> I guess it trapped on creating denormals. But again, presumably the
> reason the student used doubles here was because he wanted his
> program to be fast. Had he read just a little bit about floating
> point, he would have known that it is *not* fast under certain
Well, no. Close, but no cigar.
(a) It wasn't denormals, it was underflow.
(b) The fact underflow was handled by trapping to the operating system,
which then completed the operating by writing a 0.0 to the
register, is *NOT* a universal property of floating point, and
a universal property of IEEE floating point. It's a fact about
particular architecture, and I happened to have the manual and
(c) x*x => 0 when x is small enough *is* fast on a lot of machines.
> As it were, he seems to have applied what he though was an
> optimisation (using floating point) without knowing anything about
> it. A professional programmer would get (almost) no sympathy in
> such a situation.
You must be joking. Almost everybody working with neural nets uses
point. (One exception I came across was some people using special
processor hardware that didn't *have* floating point. These days,
probably use a programmable GPU to good effect.)
For neural net calculations, you have to do lots of dot products.
When this example happened, machines were commonly 32 bit, not 64 bit.
So doing the calculations in integers would
(a) have limited him to 16 bits for the coefficients,
instead of double's 53. This might just have been enough of a
to prevent learning the kinds of things he wanted his net to
Actually, if you want to do a dot product on n-vectors, you need
enough bits for n as well. Suppose you have 100 inputs, then
need 7 bits for that, so you are limited to (31-7)/2 = 12 bits,
is dangerously low. (doubles can do precise sums of up to 128
of coefficients with up to 23 bits).
(b) integer multiplication was very slow on that machine. On most
machines, integer multiplication is quite slow compared with add,
because architects look at C benchmarks and conclude that
isn't important. So programmers learn to do multiply-heavy
in floating point, so the benchmarks show less integer
so the architects let integer multiply get relatively slower,
(c) the sigmoid function *has* to be done in floating point
In fact I *wanted* him to use integer operations so that he could
exploit the new-at-the-time graphics instructions (think MMX),
project foundered on this step. He couldn't get a workable
of the sigmoid function using integers that didn't kill
(d) Much of the calculation needed for neural nets can be done
Basic Linear Algebra Subroutines, which are available in seriously
tuned form for most modern machines. If a programmer *didn't* use
these (floating-point-only) libraries, I would be asking why not.
If you are aware of any neural net software for general purpose
by programmers you consider competent that *doesn't* use floating
would be interested to hear about it.
More information about the Haskell-Cafe