We are running into a bit of a problem that we believe is caused by simply calling too many Kinvey requests from the JavaScript/HTML5 library at the same time. The results are inconsistent and hard to reproduce, sometimes the failed requests come back with a cross-origin header denials.



We would like to figure out a better and more optimized way to do our queries though. Let's use the following example:

-You have collection of Place (name, id) - say 10 rows

-You have collection of Listing (id,name,place,date) - place is the kinveyRef

-You have collection of ListingProperties (id,name,value,listing) - listing is the kinveyRef, value is *number*



So say you want to get the latest listing (using the date) for each of the 10 places. This would be a combination of limit,sort,group, and relations. How could you do this in a Kinvey HTML5 query? And further along, what if we wanted to join ListingProperties as well and group those with some aggregation like a sum. So in the end you would be expecting a result containing: 10 records the Listing results, and an additional property with a grouping function results of associated ListingProperties.



Is this taking the capabilities of the HTML5 library way too far? Would these situations just be better suited for a custom endpoint, or is there (could be) be a method in the HTML5 library to manually construct REST calls by specifying full URL, method, payload, etc?

This is tricky. I would use a Custom Endpoint for this, as in terms of performance it is better to move the complexity of this query to the server.



Even then, it is not straightforward. Getting the latest listings of all 10 places is possible by grouping. However, as far as I can see, this would require walking through *all* listings. Code would be similar to:



function onRequest(request, response, modules) {

var ca = modules.collectionAccess;

var col = ca.collection('listings');

col.group(['place._id'], {}, { }, function (obj, prev) {

// If place has no listing yet, or a newer one: replace.

if(null == prev.date || obj.date > prev.date) {

prev._id = obj._id;

prev.name = obj.name;

prev.date = obj.date;

}

return prev;

}, function(err, res) {

response.body = res;

response.complete();

});

}



The response body will be similar to:



[

{

"_id": "listing _id",

"place._id": "52cdba0f828edc1e1f60911a",

"name": "listing name",

"date": "most recent listing date for this place._id"

},

{

"_id": "listing _id",

"place._id": "52cdba15915560137296545d",

"name": "listing name",

"date": "most recent listing date for this place._id"

}

]



This gives you the listing data. You can now proceed either in the custom endpoint, or return this to your app and use the HTML5 library for the remainder of the work.



Next is joining the listing properties. Since you have the listing `_id`s in the response above, you’d simply do a query on *ListingProperties* where `listing._id` in `[ 'list', 'of', 'all', '_ids', 'retrieved', 'agove' ]`.



This gives you everything you need. You have the 10 places, 10 listing results, and a list of properties for these 10 listing results. You will have to manually join these results, as resolving references through the library will have no effect when grouping.

Hi Mark.



This was incredibly helpful, thanks much. I think I figured out how to make this work in HTML5 library. One of the missing parts was the *in* statement, which can be done with the contains method in the HTML5 library. The other part was the custom grouping function, which I found by creating my own Kinvey.Group method I could make this work. See example below (derived from your Kinvey.Group.count method. Would you consider it acceptable to create custom grouping methods this way? Just wondering for future reference more than anything now.



```

Kinvey.Group.latest = function (field) {

var agg = new Kinvey.Group();

if(null != field) {

agg.by(field);

}

agg.initial({ // I don't think this is required, but Kinvey library requires it or will return an error

result: 0

});

agg.reduce(function(doc, out) {

if(!out.result || (doc.date > (out.result.date || 0))) {

out.result = doc;

}

});

return agg;

}

```

Glad you got it working.



I’d recommend going with a Custom Endpoint, as we are looking to refactor the grouping functionality in our libraries, so what works now might require changes in the near future.

My concern with using a custom endpoint is the timeout duration. Since we're having to hit two collections (one waiting for the other) and walk through all records I can see this causing problems as the number of records grow. I'll see what I can work out.