Vue, Observables and large data sets

This is short example about Observables and how they can be really evil. Fortunately we have a hero called Object.freeze(obj).

Problem

Same as in previous article about Web Workers, we have large data sets, several objects that contain arrays of thousands values. This time it's not about processing, sorting, filtering these arrays, but just about storing it to vuex store. Structure looks like this:

// pseudocode
cars: {
  car1: {
    visible: Boolean(),
    metrics: Object(),
    trails: Array(30),
    values: Array(8000)[
      {available: Number, lat: String, lng: String},
      ...
    ]
  },
  ...
},
...

Execution of setter for cars state.cars = cars took around ~500 ms, which is quire a lot, when web becomes unresponsive for half second. And JS heap jumped to 270 MB, nobody wants that.

[field_image:0|center]

We can see, most of execution time is taken by reactiveSetter executed from setCars.What can I do with this? I know data won't change, it's loaded from API, and if it refreshes, whole car object gets refreshed. How to prevent those arrays to be observable?

Solution

It's simple, just get rid of those expensive Observables. You could just do this

commit('setCars', Object.freeze(cars))

[field_image:1|center]

Now we are down from 500 ms to 0.15 ms, ~3000 times faster and with JS Heap dropped to ~67 MB. But there is little problem. We have key visible, that needs to be changeable. And that won't work with whole Object frozen.

TypeError: Cannot assign to read only property 'visible' of object '#<Object>'

So we freeze just parts of Object we need.

const frozenCars = {}
Object.keys(cars).forEach(key => {
  frozenCars[key] = {
    ...cars[key],
    metrics: Object.freeze(cars[key].metrics),
    trail: Object.freeze(cars[key].trail),
    values: Object.freeze(cars[key].values)
  }
})
commit('setCars', frozenCars)

[field_image:2|center]

Ok, now we've added a bit of complexity, forEeach takes ~4 ms + 0.15 ms setCars, but still compared to 500 ms, we are good, 100 times faster and with interactive functionality.

Conclusion

Save time and memory with Object.freeze(obj), not everything needs to be observable.

Notes

You might wonder, why not to freeze data when we are transforming it (in Web Worker) and save those ~4 ms in forEach.

cars[car.metric.name] = {
  trail: Object.freeze(newCar.trail),
  visible: newCar.visible,
  metric: Object.freeze(newCar.metric),
  values: Object.freeze(newCar.values)
}

This would work nicely, if we had this on main thread. But we are sending data from worker to main with postMessage which uses Structured clone algorithm, so it would "unfreeze" objects.

You can find non-frozen code on github ra100/carshare-exporter-viz example/nofreeze branch.