[Haskell-beginners] Nonsensical behavior with http-conduit, httpLbs

[Christopher Allen]

Yeap, it was elasticsearch.

Had to call
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-refresh.htmlto
force update on indexing of documents.

Sorry to bother everybody ;_;

Thank you so much for your help.

Have a great day :)

--- Chris



On Thu, Apr 10, 2014 at 1:28 AM, Michael Snoyman <michael at snoyman.com>wrote:

> http-conduit does not perform any kind of automatic concurrency. httpLbs
> will only return after it has fully read the response body from the server.
> So unless there's a bug somewhere in http-conduit, it seems that the server
> has not fully updated its index by the time it gives back a response.
>
>
> On Thu, Apr 10, 2014 at 9:25 AM, Christopher Allen <cma at bitemyapp.com>wrote:
>
>> the function actually running the requests is:
>>
>> dispatch :: String -> Method -> Maybe L.ByteString
>>             -> IO Reply
>> dispatch url method body = do
>>   initReq <- parseUrl url
>>   let reqBody = RequestBodyLBS $ fromMaybe emptyBody body
>>   let req = initReq { method = method
>>                     , requestBody = reqBody
>>                     , checkStatus = \_ _ _ -> Nothing}
>>   withManager $ httpLbs req
>>
>> rest of the code is here: https://github.com/bitemyapp/bloodhound/
>>
>>
>> On Thu, Apr 10, 2014 at 1:25 AM, Christopher Allen <cma at bitemyapp.com>wrote:
>>
>>> It's a step removed from that, and `seq`'ing insertData isn't fixing it,
>>> only a threadDelay is.
>>>
>>> The operations insertData is performing are:
>>> delete index, create index, add document to index
>>>
>>> the operation queryTweet is performing is:
>>> search index
>>>
>>> The behavior seems to indicate "search index" is happening before "add
>>> document to index" and after "delete index", as the document will not exist
>>> when the query is performed.
>>>
>>> Only adding a threadDelay as reliably worked. Even `seq`'ing insertData
>>> doesn't work.
>>>
>>> Are the requests somehow being performed concurrently?
>>>
>>>
>>>
>>> On Thu, Apr 10, 2014 at 1:15 AM, Michael Snoyman <michael at snoyman.com>wrote:
>>>
>>>> I can't say I fully understand your code, but it seems like it's doing
>>>> the following:
>>>>
>>>> 1. Send request to server to delete data.
>>>> 2. Send request to server to add data.
>>>> 3. Request data from server.
>>>>
>>>> You're saying that with a long enough delay between steps 2 and 3, then
>>>> (3) works, otherwise it fails. It sounds to me like there's some kind of a
>>>> race condition. I don't know anything about createExampleIndex, but are you
>>>> certain that it only returns after the data is completely ready for
>>>> querying?
>>>>
>>>>
>>>> On Thu, Apr 10, 2014 at 8:43 AM, Christopher Allen <cma at bitemyapp.com>wrote:
>>>>
>>>>> to add to my question adding a delay made it work:
>>>>>
>>>>> main :: IO ()
>>>>> main = do
>>>>>   _ <- insertData
>>>>>   threadDelay 1000000
>>>>>   myTweet <- queryTweet
>>>>>   print myTweet
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 10, 2014 at 12:41 AM, Christopher Allen <cma at bitemyapp.com
>>>>> > wrote:
>>>>>
>>>>>> Repro'd on GHC 7.8.1 and 7.6.3
>>>>>>
>>>>>> Original code is: https://github.com/bitemyapp/bloodhound/
>>>>>>
>>>>>> The code only works if I manually (out of band) run insertData (in a
>>>>>> REPL) and then run main with the insertData invocation stripped out. If I
>>>>>> run main with the insertData invocation included, it throws an exception
>>>>>> (head) because the search results are empty.
>>>>>>
>>>>>> The behavior is as if queryTweet was executing after insertData
>>>>>> deleted the index, but before it inserted the new data.
>>>>>>
>>>>>> The following is a stripped down example:
>>>>>>
>>>>>> insertData :: IO ()
>>>>>> insertData = do
>>>>>>   let encoded = encode exampleTweet
>>>>>>   _ <- deleteExampleIndex
>>>>>>   created <- createExampleIndex
>>>>>>   docCreated <- indexDocument (Server "http://localhost:9200")
>>>>>> "twitter" "tweet" exampleTweet "1"
>>>>>>   print "test"
>>>>>>   return ()
>>>>>> el
>>>>>> queryTweet :: IO (Either String Tweet)
>>>>>> queryTweet = do
>>>>>>   let queryFilter = BoolFilter (MustMatch (Term "user" "bitemyapp")
>>>>>> False)
>>>>>>                     <||> IdentityFilter
>>>>>>   let search = Search Nothing (Just queryFilter)
>>>>>>   reply <- searchByIndex testServer "twitter" search
>>>>>>   let result = eitherDecode (responseBody reply) :: Either String
>>>>>> (SearchResult Tweet)
>>>>>>   let myTweet = fmap (hitSource . head . hits . searchHits) result
>>>>>>   return myTweet
>>>>>>
>>>>>> main :: IO ()
>>>>>> main = do
>>>>>>   _ <- insertData
>>>>>>   myTweet <- queryTweet
>>>>>>   print myTweet
>>>>>>
>>>>>> further up the call chain, the http call is getting dispatched with
>>>>>> http-conduit's gear, the result returned with the expression:
>>>>>>
>>>>>> withManager $ httpLbs req
>>>>>>
>>>>>> Included this in case it affects the semantics.
>>>>>>
>>>>>> Can anybody help? This has stumped me for a couple hours and I
>>>>>> couldn't get anything clueful on IRC.
>>>>>>
>>>>>> Thanks to any that help.
>>>>>>
>>>>>> --- Chris
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Beginners mailing list
>>>>> Beginners at haskell.org
>>>>> http://www.haskell.org/mailman/listinfo/beginners
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Beginners mailing list
>>>> Beginners at haskell.org
>>>> http://www.haskell.org/mailman/listinfo/beginners
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Beginners mailing list
>> Beginners at haskell.org
>> http://www.haskell.org/mailman/listinfo/beginners
>>
>>
>
> _______________________________________________
> Beginners mailing list
> Beginners at haskell.org
> http://www.haskell.org/mailman/listinfo/beginners
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/beginners/attachments/20140410/84c5fd5e/attachment.html>