Splitting SYB from the base package in GHC 6.10

[Claus Reinke]

> The issue is: SYB is being moved out of base into its own package.
> However, the Data class is, in a way, tied to base since it depends on the
> deriving mechanism.

My understanding is that the deriving mechanism would still work if
class 'Data' was moved into 'syb', but changes in 'Data' would still 
need to be matched in the deriving mechanism (which isn't auto-generated
from 'base', either). As long as 'syb' remains a core library, we can thus
focus on assigning modules to 'syb' or 'base' by functionality.

> Therefore, it was suggested that the entire Data.Generics.Basics module [2]
> should remain in base. This module defines the Data class and several
> associated functions and datatypes. I don't think anyone objected to this so
> far: please correct me if I'm wrong, or object now.

Assuming this is based on 'Data.Generics.Basics' and 'Data.Typeable'
being of more general use than the rest of 'syb' (justifying a preferred
dependency on 'base' rather than 'syb'), not any implementation constraints, 
I don't object in general. It does suggest a separate 'data-reflect' package 
for these two modules, but that could be left for later.

However, if 'Data' is in 'base', and the 'data' types are in 'base', then the 
'Data' instances for those 'data' types should probably also be in base (*)
(the instance for 'Array a b' ought to move to 'array'). And the short-term 
issue with this is that these instances, their location, and their importers, 
need some revision, while 'base' wants to be stable. 

The hope was that splitting off 'syb' from 'base' would contain the changes 
in a package with named maintainer, outside 'base'. Wouldn't it be easier 
to have all of 'Data' in 'syb', at least until 'Data' and 'Typeable' move into 
their own package? But if you can find a way to make the 'Data'-in-'base'
route work, I'm not going to object.

> Then it was also suggested that Data.Generics.Instances [3] could stay in
> base (perhaps inside Basics as well). This, however, would prevent dealing
> with the "dubious" Data instances [4], and this was one of the motivating
> factors to split SYB from base. This refers concretely to the instances:

Rearranging the list slightly, for easier reference:

-- these have (or produce) substructures of type 'a', which aren't
-- traversed by the current Data instances (contrary to what one
-- would expect, say, from a generic 'fmap' over these types)

> instance (Data a, Data b) => Data (b -> a)
> instance Typeable a => Data (IO a)
> instance (Typeable s, Typeable a) => Data (ST s a)
> instance Typeable a => Data (STM a)
> instance Typeable a => Data (IORef a)
> instance Typeable a => Data (TVar a)
> instance Typeable a => Data (MVar a)

-- here, the 'a' is a phantom type, without matching substructures
> instance Typeable a => Data (Ptr a)
> instance Typeable a => Data (StablePtr a)
> instance Typeable a => Data (ForeignPtr a)

-- here, the 'a' corresponds to substructures that should only
-- be visible through the abstract interface, on top of which a
-- 'data'-like view can be provided
> instance (Data a, Integral a) => Data (Ratio a)
 
In addition, a longer list of instances offer only runtime errors
for some 'Data' operations (most notably for 'gunfold', though
abstract types in general have a problem with reflection support). 
Are these necessary or would they profit from closer investigation?
If the latter, those instances should probably not be in 'base'.

> These instances are defined in such a way that they do not traverse the
> datatype. In fact, there is no other possible implementation, and this
> implementation at least allows for datatypes which contain both "regular"
> and "dubious" elements to still have their "regular" elements traversed.

Well, there are alternative instances that would at least improve traversal
support [3], but that wouldn't work for queries, I think.

> However, this implies that a user cannot redefine such instances even in the
> case where s/he knows extra information about these types that would allow
> for a more useful instance definition, for instance.

Indeed, the implicit presence of these instances is the main issue, and 
reducing their presence and propagation affects 'base' and other core
and extra libaries, so needs to happen soon.
 
> Claus, please correct me if I'm wrong, but if the 11 "dubious" instances (or
> perhaps less, given your message in [5]) go in the syb package and the
> remaining, "standard" ones stay in base, we:
> - Mantain backwards compatibility regarding SYB in 6.10, and
> - Can still deal with the issue by releasing a new version of the syb
> package later, independently of GHC.

issues to consider, of the top of my head:

- to what extent can core libraries be updated independent of 'base'?
- unless 'base' can now be updated (there are two versions of 'base' in
    ghc head), 'base' must not depend on 'syb'
- which other core libraries depend on 'syb'? are they updateable?
- the current importers of (parts of) 'Data.Generics' need to be revised [1]
- instances cannot be deprecated
- since all instances are in one module, one could deprecate the module,
    but are module deprecations propagated to their importers automatically?
- would 'Data.Generics' need to be deprecated, in favour of a version that
    does not implicitly re-export any/all instances? [2]

Maintaining strict backwards-compatibility in 6.10 while still allowing
for changes in 'syb' is going to be difficult, if only because clients might
depend on 'Data.IntSet' and the like to re-export all current 'Data' 
instances, which we certainly want to stop. 

My 'syb-utils' [2] has alternatives to 'Data.Generics' that export either 
only standard instances or no instances, which would allow to deprecate 
all 'Data.Generics*' modules that are less specific about their instance 
exports, but would require use of alternative module names..

> Since the deadline for 6.10 is approaching I'm assuming that we should try
> to minimize the changes there, while keeping future development in the syb
> package as open as possible.

Definitely. But some choices need to be made now. Mainly what goes
where, how to handle deprecation, and how to reduce implicit instance
propagation.

Claus

[1] http://article.gmane.org/gmane.comp.lang.haskell.libraries/9957
[2] http://www.cs.kent.ac.uk/~cr3/toolbox/haskell/#syb-utils
[3] http://www.haskell.org/pipermail/libraries/2008-July/010319.html
(*) this isn't a firm rule, either: recently, it was decided to keep the
    'Data' instances for 'ghc' types out of 'ghc'..