[Haskell-cafe] Correct way to "catch all exceptions"

[John Lato]

Hi Michael,

I don't think those are particularly niche cases, but I still think this is
a bad approach to solving the problem.  My reply to Erik explicitly covers
the worker thread case, and for running arbitrary user code (as in your top
line) it's even simpler: just fork a new thread for the user code.  You can
use the async package or similar to wrap this, so it doesn't even add any
LOCs.

What I think is particularly niche is not being able to afford the cost of
another fork, but I strongly doubt that's the case for Warp.

The reason I think this is a bad design is twofold: first maintaining a
list of exclusions like this (whether it's consolidated in a function or
part of the exception instance) seems rather error-prone and increases the
maintenance burden for very little benefit IMHO.

Besides, it's still not correct.  What if you're running arbitrary user
code that forks its own threads?  Then that code's main thread could get a
BlockedIndefinitelyOnMVar exception that really shouldn't escape the user
code, but with this approach it'll kill your worker thread anyway.  Or even
malicious/brain-damaged code that does myThreadId >>= killThread?

I like Ertugrul's suggestion though.  It wouldn't fix this issue, but it
would add a lot more flexibility to exceptions.


On Wed, Jul 10, 2013 at 6:44 PM, Michael Snoyman <michael at snoyman.com>wrote:

>
>
>
> On Wed, Jul 10, 2013 at 1:01 PM, John Lato <jwlato at gmail.com> wrote:
>
>> On Wed, Jul 10, 2013 at 5:02 PM, Erik Hesselink <hesselink at gmail.com>wrote:
>>
>>> On Wed, Jul 10, 2013 at 10:39 AM, John Lato <jwlato at gmail.com> wrote:
>>> > I think 'shouldBeCaught' is more often than not the wrong thing.  A
>>> > whitelist of exceptions you're prepared to handle makes much more
>>> sense than
>>> > excluding certain operations.  Some common whitelists, e.g. filesystem
>>> > exceptions or network exceptions, might be useful to have.
>>>
>>> You'd think that, but there are common use cases. For example, if you
>>> have a queue of work items, and a thread (or threads) processing them,
>>> it is useful to catch all exceptions of these threads. You can then
>>> log the exception, remove the item from the queue and put it in some
>>> error bucket, and continue on to the next item. The same goes for e.g.
>>> socket listening threads etc.
>>>
>>> The thing here is that you are *not* actually handling the specific
>>> exception, but instead failing gracefully. But you still want to be
>>> able to kill the worker threads, and you don't want to handle
>>> exceptions that you cannot recover from even by moving on to the next
>>> work item.
>>>
>>
>> I think that's a particularly niche use case.  We have some similar code,
>> and our approach is to have the thread re-throw (or terminate) after
>> logging the exception.  There's a separate thread that monitors the thread
>> pool, and when threads die new ones are spawned to take their place (unless
>> the thread pool is shutting down, of course).  Spawning a new thread only
>> happens on an exception and it's cheap anyway, so there's no performance
>> issue.
>>
>> As Haskell currently stands trying to sort out thread-control and
>> fatal-for-real exceptions from other exceptions seems rather fiddly,
>> unreliable, and prone to change between versions, so I think it's best
>> avoided.  If there were a standard library function to do it I might use
>> it, but I wouldn't want to maintain it.
>>
>>
> Maybe I'm just always working on niche cases then, because I run into this
> problem fairly regularly. Almost any time you want to write a library that
> will run code it doesn't entirely trust, this situation arises. Examples
> include:
>
>    - Writing a web server (like Warp) which can run arbitrary user code.
>    Warp must fail gracefully if the user code throws an exception, without
>    bringing down the entire server thread.
>    - Writing some kind of batch processing job which uses any library
>    which may throw an exception. A white list approach would not be sufficient
>    here, since we want to be certain that any custom exception types have been
>    caught.
>    - A system which uses worker threads to do much of its work. You want
>    to make certain the worker threads don't unexpectedly die because some
>    exception was thrown that you were not aware could be thrown. I use this
>    technique extensively in Keter, and in fact some work I'm doing on that
>    code base now is what triggered this email.
>
> I think that, overall, Ertugrul's suggestion is probably the right one: we
> should be including richer information in the `Exception` typeclass so that
> there's no guessing involved, and any custom exception types can explicitly
> state what their recovery preference is. In the meanwhile, I think we could
> get pretty far by hard-coding some rules about standard exception types,
> and making an assumption about all custom exception types (e.g., they *
> should* be caught by a "catch all exceptions" call).
>
> If we combine these two ideas, we could have a new package on Hackage
> which defines the right set of tags and provides a `tagsOf` function which
> works on any instance of Exception, which uses the assumptions I mentioned
> in the previous paragraph. If it's then decided that this is generally
> useful enough to be included in the Exception typeclass, we have a
> straightforward migration path:
>
>    1. Add the new method to the Exception typeclass, with a default
>    implementation that conforms with our assumptions.
>    2. For any of the special standard exception types (e.g.,
>    AsyncException), override that default implementation.
>    3. Modify the external package to simply re-export the new method when
>    using newer versions of base, using conditional compilation.
>    4. Any code written against that external package would work with both
>    current and future versions of base.
>    5. The only incompatibility would be if someone writes code which
>    overrides the typeclass method; that code would only work with newer bases,
>    not current ones.
>
> Any thoughts on this? I'm not sure exactly what would be the right method
> to add to the Exception typeclass, but if we can come to consensus on that
> and there are no major objections to my separate package proposal, I think
> this would be something moving forward on, including a library proposal.
>
> Michael
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130711/f803a494/attachment-0001.htm>