1

My Angular (v1.2.15) application needs to process a large (60k entries) index in Elasticsearch (v2.4). The index is too large to use traditional paging using from and size parameters in the Elasticsearch query. Instead I'm using the Scroll Api, as recommended.

I can get this working as expected using an HTTP client (Postman). The call sequence is as follows:

POST https://example.com/myindex/mytype/_search?scroll=1m
{
    "query":{
        "term":{
            "someFlag":false
        }
    },
    "sort":["_doc"]
}

This returns:

{
  "_scroll_id": "$$28$$VLgIOB3qj91KEA5az5jSwA==$$QXix74BkZ7CRjjBL54dy2gnVav8=cXVlcnlBbmRGZXRjaDsxOzc2MDI5OlJyQzFUX0oxUkV1aW9XclU4M2RPOFE7MDs=",
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 10002,
    "max_score": null,
    "hits": [
        ...
    ]
  }
}

Using the _scroll_id value in the next call:

POST https://example.com/myindex/mytype/_search?scroll=1m
{
    "scroll":"5m",
    "scroll_id":"$$28$$VLgIOB3qj91KEA5az5jSwA==$$xfMhpKpO0ZUjhlY7gI3Yt__DIeY=cXVlcnlBbmRGZXRjaDsxOzc2MDMyOlJyQzFUX0oxUkV1aW9XclU4M2RPOFE7MDs="
}

Returns:

{
  "_scroll_id": "$$28$$VLgIOB3qj91KEA5az5jSwA==$$xfMhpKpO0ZUjhlY7gI3Yt__DIeY=cXVlcnlBbmRGZXRjaDsxOzc2MDMyOlJyQzFUX0oxUkV1aW9XclU4M2RPOFE7MDs=",
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 10002,
    "max_score": null,
    "hits": [
        ...
    ]
  }
}

as expected.

However, when I attempt to make the equivalent calls using Angular's $http service, the first call succeeds, but no HTTP call appears to be made on subsequent calls.

var getSavedSearchIds = function () {
    var url = configService.getElasticUri() + "/_search?scroll=1m";
    var config = {
        headers: {
            "Authorization": "Basic " + configService.getElasticAuthorisationData(),
            "Content-Type": "application/json"
        }
    };
    var query = {
        query: {
            term: {
                someFlag: false
            }
        },
        sort: ["_doc"]
    };
    return $http.post(url, query, config).then(function (data) {
        return data;
    });
}

var getNextSavedSearchIds = function (scrollId) {
    var url = configService.taskElasticUri + "_search/scroll";
    var config = {
        headers: {
            "Authorization": "Basic " + configService.getElasticAuthorisationData(),
            "Content-Type": "application/json"
        }
    };
    var query = {
        scroll: "1m",
        scroll_id: scrollId
    };
    return $http.post(url, query, config).then(function (data) {
        return data;
    });
}

According to the network monitor in chrome developer tools, no HTTP request is made for the second and subsequent calls.

This looks like an Angular issue, but I haven't been able to find a solution to it, or information about it.

How can I get requests for data pages to succeed after the initial call?

NB. Angular 1.2.15 is a constraint on this solution. It can't be changed because of potential impacts in the broader application scenario.

UPDATE

The code that calls these two methods looks like this:

savedSearchService.getSavedSearchIds()
    .then(function (batch) {
        var scrollId = batch.scrollId;
        var ids = batch.Ids
        while (ids.length > 0) {
            ...
            savedSearchService.getNextSavedSearchIds(scrollId)
                .then(function (batch) {
                    scrollId = batch.scrollId;
                    ids = batch.Ids
                    ...
                });
        }
        return;
    })
5
  • Could you use the elasticsearch client rather than use $http to interact with elasticsearch? Commented Oct 27, 2016 at 12:28
  • Also, could you show how are you calling getSavedSearchIds and getNextSavedSearchIds? Commented Oct 27, 2016 at 12:29
  • Since the first request went through without any issue's it seems more like an implementation issue than an angular or ES issue. Try setting breakpoints at the post calls in the developer tools to see if they are actually being called. Commented Oct 27, 2016 at 12:45
  • @adityasinghraghav yes, the post call is being called, but a breakpoint in the then() callback is never hit. Commented Oct 27, 2016 at 13:22
  • @pedromarce Unfortunately, this project doesn't use the elasticsearch client, and to add it would be to introduce another dependency. I'll update the post with the calling code as suggested. Commented Oct 27, 2016 at 13:25

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.