[web-devel] Re: New Persistent Meta-Backend for Horizontal Partitioning - Difficulty?

[Max Cantor]

Forgot to mention, one other advantage of this system would be that the backends could be mixed.  For instance, you could have an in-memory postgres DB backend on the same server as your yesod server (I think I heard someone mention that they are developing one) for a smaller set of low-latency data and a large, postgres database on another server which doesn't need to be solely in memory.

Just another thought.

max

On Nov 2, 2010, at 1:21 PM, Max Cantor wrote:

> I like Persistent for its simplicity and ease of use compared to the other, more full-featured DB libraries on Hackage.  There is one feature which I am interested in but am curious what the community thinks of it.  
> 
> My idea is to support horizontal partitioning via a new Persistent backend which is more of a meta backend that redirects queries to any number of child backends.  I think it shouldn't be too difficult.  Persistent's simplicity and not supporting joins makes this much easier.  The meta-backend simply needs have references to the child back ends and some sort of "chooser" function that says which data goes where.  Queries which need to be executed across the children could run in parallel (with some restrictions on things like sorting).  
> 
> Anyway, just a thought.  I'm curious if this community thinks this would be useful given that tools like pgpool exist.  I've heard some bad things about pgpool which is why I was thinking about this.  Furthermore, glancing at the code for the current backends, it doesn't *seem* like a ton of work, but I am far from convinced.
> 
> So, what does everything thing?