No subject
Tue Feb 1 14:29:29 CET 2011
For example i have two datasets "A" and "B".
I want join its by some rule and take only first "N" results. In this case
i do not want serve all datasets entirely. I would like to make as little a=
s
possible.
I think this algorithms are exist.
2011/2/22 Max Cantor <mxcantor at gmail.com>
> I strongly disagree.
>
> Because Persistent doesn't do joins, it happens to be a good candidate fo=
r
> a "web-scale" horizontally sharded solution, which I'll get to in a minut=
e.
>
> Its perfectly easy to perform joins on the app-server in haskell code
> rather than relying on the database to do them. Facebook does a lot of
> this. We do this. Note that its much easier to replicate app-servers th=
an
> DB servers.
>
> About making persistent-webscale, there are two features whose additions
> would basically accomplish this. Michael, I am thinking of offering a sm=
all
> bounty if you want to put them together.
>
> One is sharding. In the mkPersist function, if you could add a partition
> function which is somethng like: PersistEntity e =3D> e -> SqlConnection =
and
> then alter the PersistBackend instance to respect that function, you get
> sharding almost for free. Its a fair bit more complicated than I am
> describing here but I believe its very possible. joins would make this
> inordinately more complicated.
>
> Also, integrated memcached. It would be great if you could annotate tabl=
es
> to store in memcached. when you set up your connection, optionally inclu=
de
> a memcached connection. then persistent would check in each select, get,
> getBy call if the value is in memcached before going to the DB.
>
> I'm not very familiar with persistent internals so I could be wrong, but =
I
> think these shouldn't be too hard to add.
>
> </rant>
>
> Max
>
> On Feb 22, 2011, at 5:37 PM, Michael Snoyman wrote:
>
> > Hi James,
> >
> > It's funny you mention this now. I recently got an email from =D0=90=D0=
=BD=D1=82=D0=BE=D0=BD
> > =D0=A7=D0=B5=D1=88=D0=BA=D0=BE=D0=B2 about this very topic. Hopefully h=
e can share the code he sent
> > me with the list. The one improvement that we're planning on making is
> > to write it using enumerators instead of lists so that the memory
> > requirements are lower.
> >
> > For now, however, you might have noticed that the Haskellers code does
> > a number of "pseudo-joins" all over the place, which currently works
> > well enough. =D0=90=D0=BD=D1=82=D0=BE=D0=BD's suggested code basically =
automates a lot of this
> > process.
> >
> > Michael
> >
> > On Tue, Feb 22, 2011 at 11:25 AM, James <cnjamesdeng at gmail.com> wrote:
> >> Hi all,
> >>
> >> I'm new to this mailing list. I did web development in python with
> >> Django framework before. But I really like the haskell language, so
> >> considering building my next site (a LBS-SN like service) in haskell. =
I
> >> learn snap for a while and think it's not so mature and powerful. Afte=
r
> >> I learned the yesod docs in one day and read some yesod and
> >> haskellers.com codes in the following few days, I appreciate it and tr=
y
> >> to use it seriously.
> >>
> >> Now I need a database query across two table, so as to sql join featur=
e.
> >> Currently, persistent doesn't provide this functionality. I think mayb=
e
> >> adding a function to PersistBackend class, and implementing it in
> >> PersistBackend (SqlPersist m) may solve the problem. However, I am not
> >> sured. Will it be difficult and consistent to implement this feature?
> >>
> >> Thanks for any help.
> >>
> >> Sincerely,
> >> --
> >> James Deng
> >> department of computer science,
> >> school of information science & technology,
> >> Sun-yat-sen University, Guangzhou, China
> >> mailTo: cnJamesDeng at gmail.com
> >> homepage: http://cnjdeng.appspot.com
> >>
> >>
> >> _______________________________________________
> >> web-devel mailing list
> >> web-devel at haskell.org
> >> http://www.haskell.org/mailman/listinfo/web-devel
> >>
> >
> > _______________________________________________
> > web-devel mailing list
> > web-devel at haskell.org
> > http://www.haskell.org/mailman/listinfo/web-devel
>
>
> _______________________________________________
> web-devel mailing list
> web-devel at haskell.org
> http://www.haskell.org/mailman/listinfo/web-devel
>
--=20
Best regards,
Cheshkov Anton
Phone: +7 909 005 18 82
Phone: +7 931 511 47 37
Skype: cheshkov_anton
--000e0cd328b2caf01f049cdd27f6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
We all =C2=A0want to achieve the same goal =C2=A0 make data join in applica=
tion layer. =C2=A0<div>From my point of view the main problem is effective =
scale join algorithm.=C2=A0</div><div>For example i have two datasets =C2=
=A0"A" =C2=A0and "B".</div>
<div>I want join =C2=A0its by some rule and take only first "N" r=
esults. In this case i do not want serve all datasets=C2=A0<span class=3D"A=
pple-style-span" style=3D"font-family: arial, sans-serif; font-size: 14px; =
white-space: nowrap; ">entirely. I would like to make as little as possible=
.</span>=C2=A0=C2=A0</div>
<div>I think this=C2=A0algorithms=C2=A0=C2=A0are exist. =C2=A0=C2=A0</div><=
meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><di=
v><br><div class=3D"gmail_quote">2011/2/22 Max Cantor <span dir=3D"ltr"><=
;<a href=3D"mailto:mxcantor at gmail.com">mxcantor at gmail.com</a>></span><br=
>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">I strongly disagree.<br>
<br>
Because Persistent doesn't do joins, it happens to be a good candidate =
for a "web-scale" horizontally sharded solution, which I'll g=
et to in a minute.<br>
<br>
Its perfectly easy to perform joins on the app-server in haskell code rathe=
r than relying on the database to do them. =C2=A0Facebook does a lot of thi=
s. =C2=A0We do this. =C2=A0Note that its much easier to replicate app-serve=
rs than DB servers.<br>
<br>
About making persistent-webscale, there are two features whose additions wo=
uld basically accomplish this. =C2=A0Michael, I am thinking of offering a s=
mall bounty if you want to put them together.<br>
<br>
One is sharding. =C2=A0In the mkPersist function, if you could add a partit=
ion function which is somethng like: PersistEntity e =3D> e -> SqlCon=
nection and then alter the PersistBackend instance to respect that function=
, you get sharding almost for free. =C2=A0Its a fair bit more complicated t=
han I am describing here but I believe its very possible. =C2=A0joins would=
make this inordinately more complicated.<br>
<br>
Also, integrated memcached. =C2=A0It would be great if you could annotate t=
ables to store in memcached. =C2=A0when you set up your connection, optiona=
lly include a memcached connection. then persistent would check in each sel=
ect, get, getBy call if the value is in memcached before going to the DB.<b=
r>
<br>
I'm not very familiar with persistent internals so I could be wrong, bu=
t I think these shouldn't be too hard to add.<br>
<br>
</rant><br>
<font color=3D"#888888"><br>
Max<br>
</font><div><div></div><div class=3D"h5"><br>
On Feb 22, 2011, at 5:37 PM, Michael Snoyman wrote:<br>
<br>
> Hi James,<br>
><br>
> It's funny you mention this now. I recently got an email from =D0=
=90=D0=BD=D1=82=D0=BE=D0=BD<br>
> =D0=A7=D0=B5=D1=88=D0=BA=D0=BE=D0=B2 about this very topic. Hopefully =
he can share the code he sent<br>
> me with the list. The one improvement that we're planning on makin=
g is<br>
> to write it using enumerators instead of lists so that the memory<br>
> requirements are lower.<br>
><br>
> For now, however, you might have noticed that the Haskellers code does=
<br>
> a number of "pseudo-joins" all over the place, which current=
ly works<br>
> well enough. =D0=90=D0=BD=D1=82=D0=BE=D0=BD's suggested code basic=
ally automates a lot of this<br>
> process.<br>
><br>
> Michael<br>
><br>
> On Tue, Feb 22, 2011 at 11:25 AM, James <<a href=3D"mailto:cnjamesd=
eng at gmail.com">cnjamesdeng at gmail.com</a>> wrote:<br>
>> Hi all,<br>
>><br>
>> I'm new to this mailing list. I did web development in python =
with<br>
>> Django framework before. But I really like the haskell language, s=
o<br>
>> considering building my next site (a LBS-SN like service) in haske=
ll. I<br>
>> learn snap for a while and think it's not so mature and powerf=
ul. After<br>
>> I learned the yesod docs in one day and read some yesod and<br>
>> <a href=3D"http://haskellers.com" target=3D"_blank">haskellers.com=
</a> codes in the following few days, I appreciate it and try<br>
>> to use it seriously.<br>
>><br>
>> Now I need a database query across two table, so as to sql join fe=
ature.<br>
>> Currently, persistent doesn't provide this functionality. I th=
ink maybe<br>
>> adding a function to PersistBackend class, and implementing it in<=
br>
>> PersistBackend (SqlPersist m) may solve the problem. However, I am=
not<br>
>> sured. Will it be difficult and consistent to implement this featu=
re?<br>
>><br>
>> Thanks for any help.<br>
>><br>
>> Sincerely,<br>
>> --<br>
>> James Deng<br>
>> department of computer science,<br>
>> school of information science & technology,<br>
>> Sun-yat-sen University, Guangzhou, China<br>
>> mailTo: <a href=3D"mailto:cnJamesDeng at gmail.com">cnJamesDeng at gmail=
.com</a><br>
>> homepage: <a href=3D"http://cnjdeng.appspot.com" target=3D"_blank"=
>http://cnjdeng.appspot.com</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> web-devel mailing list<br>
>> <a href=3D"mailto:web-devel at haskell.org">web-devel at haskell.org</a>=
<br>
>> <a href=3D"http://www.haskell.org/mailman/listinfo/web-devel" targ=
et=3D"_blank">http://www.haskell.org/mailman/listinfo/web-devel</a><br>
>><br>
><br>
> _______________________________________________<br>
> web-devel mailing list<br>
> <a href=3D"mailto:web-devel at haskell.org">web-devel at haskell.org</a><br>
> <a href=3D"http://www.haskell.org/mailman/listinfo/web-devel" target=
=3D"_blank">http://www.haskell.org/mailman/listinfo/web-devel</a><br>
<br>
<br>
_______________________________________________<br>
web-devel mailing list<br>
<a href=3D"mailto:web-devel at haskell.org">web-devel at haskell.org</a><br>
<a href=3D"http://www.haskell.org/mailman/listinfo/web-devel" target=3D"_bl=
ank">http://www.haskell.org/mailman/listinfo/web-devel</a><br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br>Best regard=
s,<br>Cheshkov Anton <br>Phone: +7 909 005 18 82<br>Phone: +7 931 511 47 37=
<br>Skype: cheshkov_anton<br>
</div>
--000e0cd328b2caf01f049cdd27f6--
More information about the web-devel
mailing list