<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">My thanks to you, Simon, for your prompt response. After reading your message, I realized I perhaps read too far into this sentence on the wiki page:<div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Defining type families in plugins is more work than it needs to be, because the current interface forces the plugin to search the unsolved constraints for the type family in question (which might be anywhere within the types), <b class="">then emit a new given constraint to reduce the type family</b>.<br class=""></blockquote><div class=""><br class=""></div>Emphasis mine. Although that sentence seems to imply that adding given constraints from a typechecker plugin is both possible and sanctioned, your message, my experimentation, and the GHC source code all seem to agree that is not true, after all. After realizing that was a dead end, I puzzled for some time on how one is intended to solve nested type families after all, only to realize that a proper solution is simpler than I had realized.</div><div class=""><br class=""></div><div class="">To make things more concrete, I was confused about how my plugin might solve a constraint like</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">[W] {co_0} :: F (ToLower "Foo") ~# Length "bar"</div></blockquote><div class=""><br class=""></div><div class="">if it knows about ToLower and Length but nothing about F. The solution is actually extremely straightforward to implement, but the idea was not at all obvious to me at first. Namely, it’s possible to just recursively walk the whole type and solve any known families bottom-up, producing a new constraint:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">[W] {co_1} :: F "foo" ~# 3</div></blockquote><div class=""><br class=""></div><div class="">At the same time, to construct evidence for the first constraint, the recursive function can also build a coercion as it goes, producing</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">co_2 = (F (Univ :: "foo" ~ ToLower "Foo")) ~# (Univ :: 3 ~ Length "bar")</div></blockquote><div class=""><br class=""></div><div class="">which can be used to cast the evidence for the new constraint to evidence for the old one:</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">co_0 := co_2 ; {co_1}</div></blockquote><div class=""><br class=""></div><div class="">That all seems to be working well, and it’s much nicer than whatever it was I was trying before. There’s one small wrinkle I’ve bumped into, however, which was preventing the solver from terminating. Specifically, typechecker plugins do not seem to be able to solve derived constraints.</div><div class=""><br class=""></div><div class="">To elaborate, my plugin was getting called with the constraint [D] (Length "f" ~ 1), which it was happily turning into [D] (1 ~ 1), but the derived constraint was never removed from the inert set, and it would produce new [D] (1 ~ 1) constraints forever. I went looking in the GHC source, only to discover that you, Simon, <a href="https://gitlab.haskell.org/ghc/ghc/blob/986643cb3506f2eedce96bf2d2c03873f105fad5/compiler/typecheck/TcInteract.hs#L276-277" class="">are apparently deeply suspicious</a> of typechecker plugins that solve derived constraints! I don’t know if there’s a reason behind that—maybe I’m going about things the wrong way, and I shouldn’t need to solve those derived constraints—but in the meantime, I added some kludgey mutable state to keep track of the derived constraints my plugin has already attempted to solve so it won’t try to solve them again.</div><div class=""><br class=""></div><div class="">Anyway, that aside, if anyone finds what I’ve written in this email helpful, I can certainly flesh it out a bit more and stick it in a wiki page somewhere. Though, for what it’s worth, I think it would be even more helpful if some of the relevant information made its way into the GHC user’s guide, since I found that more discoverable than the wiki page (which took a comparatively large amount of digging to locate).</div><div class=""><br class=""></div><div class="">Thanks again,</div><div class="">Alexis</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 1, 2019, at 03:01, Simon Peyton Jones <<a href="mailto:simonpj@microsoft.com" class="">simonpj@microsoft.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Georgia; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Alexis<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Thanks for writing this up so carefully. I hope that others will join in. And please then put the distilled thought onto the wiki page(s) so they are not lost.<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Some quick thoughts from me:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><ul type="disc" style="margin-bottom: 0cm; margin-top: 0cm;" class=""><li class="MsoListParagraph" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"><b class=""><span class="">Flattening</span></b><span class="">. I’m pretty sure we pass constraints unflattened because that’s what someone wanted at the time. It could easily be changed, but it might complicate the API. E.g. you might reasonably want to know the mapping from type variable to function application. There is no fundameental obstacle.<o:p class=""></o:p></span></li></ul><div style="margin: 0cm 0cm 0.0001pt 36pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><ul type="disc" style="margin-bottom: 0cm; margin-top: 0cm;" class=""><li class="MsoListParagraph" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;"><b class=""><span class="">Letting the plugin add given constraints</span></b><span class="">. This looks a bit like: let the plugin prove lemmas and hand them back to GHC (along with their proof) to exploit. Yes, that seems reasonable too. Again, something new in the API.<o:p class=""></o:p></span></li></ul><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">I don’t understand enough of your type-class instance question to comment meaningfully, but perhaps others will.<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Nothing about the plugin interface is cast in stone. There are quite a few “customers” but few enough that they’ll probably be happy to adapt to changes. Go for it, in consultation with them!<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Simon<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0cm 0cm 0cm 4pt;" class=""><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span lang="EN-US" class="">From:</span></b><span lang="EN-US" class=""><span class="Apple-converted-space"> </span>ghc-devs <<a href="mailto:ghc-devs-bounces@haskell.org" class="">ghc-devs-bounces@haskell.org</a>><span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Alexis King<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>01 August 2019 07:55<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:ghc-devs@haskell.org" class="">ghc-devs@haskell.org</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Properly writing typechecker plugins<o:p class=""></o:p></span></div></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi all,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I have recently decided to try writing a GHC typechecker plugin so I can get my hands on some extra operations on type-level strings. My plugin works, but only sort of—I know some things about it are plain wrong, and I have a sneaking suspicion that plenty of other things are not handled properly.<o:p class=""></o:p></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">First, some context about what I do and don’t already know: I have a high-level understanding of the basic concepts behind GHC’s solver. I understand what evidence is and what purpose it serves, I mostly understand the different flavors of constraints, and I think I have a decent grasp on some of the operational details of how individual passes work. I’ve spent a little time reading through comments in the GHC source code, along with pieces of the source code itself, but I’m sure my understanding is pretty patchy.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">With that out of the way, here are my questions:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><ol start="1" type="1" style="margin-bottom: 0cm;" class=""><li class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">First, I’m trying to understand: why are wanted constraints passed to typechecker plugins unflattened? This is my single biggest point of confusion. It certainly seems like the opposite of what I want. Consider that I have a type family<br class=""><br class=""> type family ToUpper (s :: Symbol) :: Symbol where {}<br class=""><br class="">that I wish to solve in my plugin. At first, I just naïvely looked through the bag of wanted constraints and looked for constraints of the shape<br class=""><br class=""> t ~ ToUpper s<br class=""><br class="">but this isn’t good enough, since my plugin regularly receives constraints that look more like<br class=""><br class=""> t ~ SomeOtherTypeFamily (ToUpper s)<br class=""><br class="">so I have to recursively grovel through every type equality constraint looking for an application of a family I care about. Furthermore, once I’ve found one, I’m not sure how to actually let GHC know that I’ve solved it—do I really have to just generate a new given constraint and let GHC’s solver connect the dots?<br class=""><br class="">I have seen <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2Fwikis%2Fplugins%2Ftype-checker%23under-discussion-defining-type-families&data=02%7C01%7Csimonpj%40microsoft.com%7C66b089f679154ccfd8a108d7164d2ce1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637002393033597031&sdata=dr9ERzE28ev%2B3R0EbajMqM166YVI5dFm2J2Jd2D2mYo%3D&reserved=0" style="color: purple; text-decoration: underline;" class="">the note on the typechecker plugins wiki page</a> about possibly baking support for type families into the plugin interface, which would indeed be nicer than the status quo, but it seems odd to me that they aren’t just passed to plugins flattened, which seems like it would spare a lot of effort. Isn’t the flattened representation really what typechecker plugins would like to see, anyway?<o:p class=""></o:p></li><li class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">But let’s put families aside for a moment. I’m not just solving type families in my plugin, I’m also solving classes. These classes have no methods, but they do have functional dependencies. For example, I have a class<br class=""><br class=""> class Append (a :: Symbol) (b :: Symbol) (c :: Symbol) | a b -> c, a c -> b, b c -> a<br class=""><br class="">which is like GHC.TypeLits.AppendSymbol, but the fundeps make GHC a bit happier when running it “backwards” (since GHC doesn’t really know about AppendSymbol’s magical injectivity, so it sometimes complains).<br class=""><br class="">In any case, I was hoping that GHC’s solver would handle the improvement afforded by the fundeps for me once I provided evidence for Append constraints, but that doesn’t seem to be the case. Currently, I am therefore manually generating derived constraints based on the functional dependency information, plumbing FunDepOrigin2 through and all. Is there some way to cooperate better with GHC’s solver so I don’t have to duplicate all that logic in my plugin?<br class=""><br class="">I guess one thing I didn’t try is returning given constraints from my solver instead of just solving them and providing evidence. That is, if my plugin received a<br class=""><br class=""> [W] d1 :: Append "foo" "bar" c<br class=""><br class="">constraint, then instead of solving the constraint directly, I could leave it alone and instead return a new constraint<br class=""><br class=""> [G] d2 :: Append "foo" "bar" "baz"<br class=""><br class="">and presumably GHC’s solver would use that constraint to improve and solve d1. But similar to my confusion about type families above, I’m uncertain if that’s the intended method or not, since it seems like it’s sort of circumventing the plugin API.<o:p class=""></o:p></li><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;">Finally, on a related note, building evidence for these solver-generated typeclass instances is a bit of a pain. They have no methods, but they do sometimes have superclasses. Currently, I’ve been generating CoreExprs as carefully as I’m able to after reading through the desugaring code: I call dataConWrapId on the result of classDataCon, then use mkTyConApp and mkCoreApps on that directly. It seems to work okay now, but it didn’t always: -dcore-lint thankfully caught my mistakes, but I’ve been wondering if there’s a safer way to build the dictionary that I’ve been missing.<o:p class=""></o:p></li></ol><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">That’s it for now—I’ve just been muddling through until things work. Once I get something that feels closer to right, maybe I’ll put the code somewhere and ask for more low-level comments if anyone would like to take the time to offer them, but for now, I’m still working on the high-level ideas. The wiki pages I’ve found have been very helpful; my appreciation to all who have contributed to them!<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Many thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Alexis</div></div></div></div></div></blockquote></div><br class=""></div></body></html>