Grepular

Meteor JS is a universal/isomorphic Javascript framework for building Node web applications. It uses MongoDB as a document store (database). You can subscribe to Mongo collections on the front-end and data is pushed up to the client over a websocket as soon as it changes in the database. They call this “reactivity”, and when it works it’s beautiful. Your web pages update in real time to show the current state of the database, rather than the traditional method of displaying a snapshot of the data as it existed at the time a page was requested.

What I want to discuss is a very simple and common use case for web applications: pagination. I want to be able to sort a dataset, and display a subset of that data on a webpage. The user can then skip to the next or previous subset, or increase/decrease the size of the subset to display. Should be simple right? Imagine I have a collection of tickets, and I want to sort them by subject line. On the server, I publish the data as follows:

Meteor.publish('tickets', (pageNum, limit) => {
  return ticketCollection.find({}, {
    limit,
    skip: pageNum * limit,
    sort: { subject: 1 }
  })
})

On the client, I subscribe to that published data and access it as follows:

const pageNum = 3;
const limit   = 10;
Meteor.subscribe('tickets', pageNum, limit, () => {
  var ticketData = ticketCollection.find({}, {
    sort: { subject: 1 }
  }).fetch()
});

Now “ticketData” automatically updates on the client side according to whatever happens to the data in MongoDB, server side. The problem I (and many others) are having is with sorting in MongoDB. Specifically given the limitations that Meteor’s MongoDB client library (Minimongo) imposes. Mongos default sorting is case sensitive. I.e, here is an example of a sorted list according to Mongo:

• “Chumbawumba”
• “Donkey”
• “Zoro”
• “alfred”

Mongo supplies something called the “aggregation framework“. You can abuse it to create a virtual lower cased version of a field and then sort on that:

ticketCollection.aggregate([
  {
    $project: {
      lc_subject: { $toLower: "$subject" },
      subject: 1
    }
  },
  {
    $sort: { lc_subject: 1 },
  }
])

The trouble with this is, Minimongo does not support aggregation. There are third party libraries which bolt it on, but you lose reactivity when using it, meaning you no longer get client side updates as they happen. Which frankly, is the main selling point of Meteor. There are other horrible workarounds. You can write a publication which returns the aggregate result, then immediately starts observing the collection for any changes and re-runs the aggregate query whenever there is a change. This will provide terrible performance for large data sets. You can add duplicate data to the database, i.e, any column which needs to be sorted in a case insensitive manner would have a duplicate lower case version in Mongo. You can even sort client side in JavaScript. But that is of no use when you only want to send a subset of the data for pagination purposes, so the sorting needs to happen server side pre-pagination. Who ever thought it would be as difficult as this to do a bit of sorting on a web page? Bring on Postgres support.