How to structure stores for both paginated and unpaginated data

[aniket-banyal]

I have this model that is used to store information about field groups.
These field groups are shown in an infinite list.

Now, there is another feature that I need to add which is of reordering the field groups.
But, for that I need to fetch and store all the field groups, while the store I use below is storing paginated data.
Also, in case of reordering i would only need id and name of the field group.

My question is what would be the right way to store both the paginated data and the full data.
Should I somehow use only 1 store for both or have separate store.
If I have a separate store having the same kind of data, will it lead to redundancy?
Would it make sense to store the actual data in one store and only references in the paginated store?

As much as I love mobx-state-tree, the problem that I have been facing ever since using it is that there are not much resources that talk about how to structure the models and stores.

export const ContactFieldGroupModel = types
  .model({
    id: types.identifierNumber,
    name: types.string,
    description: types.maybeNull(types.string),
    color: types.maybeNull(types.string),
    icon: types.maybeNull(types.string),
    created_by: types.maybeNull(UserModel),
  })

export const ContactFieldGroupStoreModel = types
  .model({
    fieldGroups: types.array(ContactFieldGroupModel),
    count: types.optional(types.number, 0),
    page: types.optional(types.number, 0),
    hasMore: types.optional(types.boolean, false),
    loading: types.optional(types.boolean, false),
  })
  .actions((self) => ({
    getFieldGroups: flow(function* (page: number, limit?: number) {
      console.log('getFieldGroups');

      const resp = yield* toGenerator(api.getContactFieldGroups({ page, limit }));
      if (resp.kind !== 'ok') {
        self.loading = false;
        return;
      }

      self.fieldGroups = resp.fieldGroups;
      self.page = page;
      self.count = resp.count;
      self.hasMore = resp.hasMore;
      self.loading = false;
    }),
    async refreshFieldGroups() {
      self.loading = true;
      await this.getFieldGroups(1);
    },
    async getMoreFieldGroups() {
      self.loading = true;
      await this.getFieldGroups(self.page + 1);
    },
  }))

[coolsoftwaretyler]

Hey @Dummy26 - thanks for your question! I don't think there's one "right" way to do this,

Since you only care about a subset of fields for reordering, you might be able to load that data without pagination, assuming you can query your database and return just the id and name fields. I would personally load that up into a separate store or model for your re-ordering feature, and then treat it as de-normalized data. Once you load that all up into the reordering feature, you can reorder the fields, write the new order of them into whatever persistent database you have, and make sure the ContactFieldGroupStoreModel has a way to invalidate its old pagination data, and re-fetch if appropriate.

I agree there aren't enough resources out there on MST patterns. I've been trying to blog about MST as much as I personally have time for, but we can always use more people chiming in. I would encourage you to evaluate the tradeoffs of my recommendation and write about it yourself to help others in the future. Perhaps there's a good reason for you to paginate through the re-ordering data, or maybe if you don't have too many ContactFieldGroups, you can stop paginating and load it all in one go.

In my day-job, we tend to oscillate between data loading strategies based on our priorities at any given time, and based on changing feature requirements. It's tough to find an approach that works across the board. Usually we pick whatever seems easiest and fastest to implement, and only change it if we run into a clearly identifiable problem.

[aniket-banyal]

I created another store for the re-order feature

const FieldGroupModel = types.model({
  id: types.identifierNumber,
  name: types.string,
});

const ContactConfigStoreModel = types
  .model({
    fieldGroups: types.array(FieldGroupModel),
    count: types.optional(types.number, 0),
    loading: types.optional(types.boolean, false),
  })
  .actions((self) => ({
    setFieldGroups: (fieldGroups) => {
      self.fieldGroups = fieldGroups;
    },
    getAllFieldGroups: flow(function* () {
      const resp = yield* toGenerator(api.getContactFieldGroups({ page: 1, limit: 1000 }));
      if (resp.kind !== 'ok') {
        self.loading = false;
        return;
      }

      self.fieldGroups = resp.fieldGroups;
      self.count = resp.count;
      self.loading = false;
    }),
    reorderFieldGroups: flow(function* () {
      const newOrder = {};
      for (let i = 0; i < self.fieldGroups.length; i += 1) {
        const fieldGroup = self.fieldGroups[i];
        newOrder[fieldGroup.id] = i;
      }

      yield* toGenerator(api.reorderContactFieldGroups({ ordering: newOrder }));
    }),
  }));

then treat it as de-normalized data

Can you explain this more.

Also can you shed some light on how to invalidate the old paginated data on re-order. Is there a mobx way to do that or do I need to manually call the refresh function of ContactFieldGroupStoreModel on successful re-order

[coolsoftwaretyler]

Hey @Dummy26, sure thing.

De-normalized data

Here's an overview of the term itself, mostly as it relates to databases. I'm using the term somewhat loosely, but I think it's relevant.

If you were to work with the data in a "normalized" way, you would pull the entire, paginated list of ContactFieldGroupModels, and work with them in your re-ordering feature. As you've already identified, reading that data will be a pain, especially since you already have it paginated, and you need to see the full picture.

Working with this data in a "de-normalized" way means that you get duplicative data (your new model which just contains the ID and name). Since that data is smaller, and probably less necessary to paginate (or at least, since it's a new model and you can choose not to paginate on the data fetching), that makes things easier to work with while reading it into your application.

The tradeoff with this de-normalized data is that once you make your changes through reorderFieldGroups, the state of your ContactFieldGroupModel is incorrect.

The article I posted uses terms like "slow reads, slow writes, fast reads, and fast writes". In this case, it's less about speed, and more about extra logic to write. The de-normalized approach will be easier to pull data in, but more complex to manage it after you're done updating the order.

Hopefully that clears things up for you, but I'm happy to dig in further. Again, "de-normalization" here is a loose application of the term. I just mean "use this data in a duplicative way for convenience".

Invalidating data

Since you're fetching the data through your MST store, you could add a boolean field to ContactConfigStoreModel like needsRefresh. At the end of each successful call to getAllFieldGroups, you would set that to false, to indicate the data is current. Then, in some kind of hook after you finish reorderFieldGroups (either using a lifecycle hook, or just some imperative code at the end, you could get to your ContactConfigStoreModel and set needsRefresh to true. Alternatively, you could call getAllFieldGroups and skip the value. It sort of depends on how you interact with these models. If you have some kind of UI code that determines when to call getAllFieldGroups, I think it makes sense to use the flag.

Data fetching recommendation

I love MobX-State-Tree, and I know a lot of people use it to fetch and store data from the server. I do that as well at my day job. But in a lot of ways, we are re-inventing the wheel. When I said "invalidate the query", I was really thinking of tanstack query. They have a quick way to invalidate your queries.

As time goes on, I'm starting to recommend people reach for a separate library to manage their asynchronous data, and load it into MST. At my workplace, we are incrementally adopting react-query, and hydrating our MST stores with it. So if I was building your feature out today, I would keep the data fetching in react-query (or whichever version makes sense for you), and then in my reorderFieldGroups action, have some kind of callback method that could call queryClient.invalidateQueries({ queryKey: ['ContactFieldGroupModel']}).

If you don't have time or interest in pulling in another dependency, ignore everything in this section and add some more information to your ContactFieldGroupStoreModel to model the network state. TanStack query is still a great place to look at for examples of the kind of information you might want a network-aware store to model.

[aniket-banyal]

Thanks @coolsoftwaretyler for you help.
Can you please share some example of integrating react-query with MST.

[coolsoftwaretyler]

Sure. Here's a really rudimentary example:

import "./styles.css";
import { useQuery, QueryClient, QueryClientProvider } from "react-query";
import { observer } from "mobx-react";
import { types } from "mobx-state-tree";

const queryClient = new QueryClient();

const asyncRequest = async () => {
  const response = await fetch("https://jsonplaceholder.typicode.com/todos");
  return response.json();
};

const Todo = types
  .model("Todo", {
    id: types.number,
    title: types.string,
    completed: types.boolean,
  })
  .views((self) => ({
    get compositeKey() {
      return `${self.id}-${self.title}`;
    },
  }));

const TodoStore = types
  .model("TodoStore", {
    todos: types.array(Todo),
  })
  .actions((self) => ({
    setTodos(todos) {
      self.todos = todos;
    },
  }));

const todoStore = TodoStore.create({ todos: [] });

const TodoComponent = observer(() => {
  useQuery("todos", asyncRequest, {
    onSuccess: (todos) => {
      todoStore.setTodos(todos);
    },
  });

  return (
    <div className="App">
      <h1>Todos</h1>
      <ul>
        {todoStore.todos.map((todo) => (
          <li key={todo.id}>
            <input type="checkbox" checked={todo.completed} readOnly />
            {todo.compositeKey}
          </li>
        ))}
      </ul>
    </div>
  );
});

export default function App() {
  return (
    <QueryClientProvider client={queryClient}>
      <TodoComponent />
    </QueryClientProvider>
  );
}

CodeSandbox if you want to mess around with it a little: https://codesandbox.io/p/sandbox/suspicious-mclaren-mkgkjg?file=%2Fsrc%2FApp.js%3A66%2C1

[aniket-banyal]

I came across mst-reference-pool which I think can help prevent duplicate data.

This is how I have used it

const ContactFieldGroupStoreModel = types
  .model({
    pool: types.array(ContactFieldGroupModel),
    fieldGroups: types.array(types.reference(ContactFieldGroupModel)),
    count: types.optional(types.number, 0),
    page: types.optional(types.number, 0),
    hasMore: types.optional(types.boolean, false),
    loading: types.optional(types.boolean, false),
  })
  .extend(withReferencePool(ContactFieldGroupModel))
  .actions((self) => ({
    getFieldGroups: flow(function* (page: number, limit?: number) {
      const resp = yield* toGenerator(api.getContactFieldGroups({ page, limit }));
      if (resp.kind === 'ok') {
        self.fieldGroups = self.addAllToPool(resp.fieldGroups);
        self.page = page;
        self.count = resp.count;
        self.hasMore = resp.hasMore;
      }
      self.loading = false;
    }),
  }));

const ContactConfigStoreModel = types
  .model({
    fieldGroups: types.array(types.reference(ContactFieldGroupModel)),
    count: types.optional(types.number, 0),
    loading: types.optional(types.boolean, false),
  })
  .actions((self) => ({
    getAllFieldGroups: flow(function* () {
      const resp = yield* toGenerator(api.getContactFieldGroups({ page: 1, limit: 1000 }));
      if (resp.kind === 'ok') {
        const fieldGroupPoolStore = getRoot(self).contactFieldGroupStore;
        self.fieldGroups = fieldGroupPoolStore.addAllToPool(resp.fieldGroups);
        self.count = resp.count;
      }

      self.loading = false;
    }),
    setFieldGroups: (fieldGroups) => {
      self.fieldGroups = fieldGroups;
    },
  }));

Now, the fieldGroups are stored only in the pool and both the stores contain references to the pool.

I have tried using mst-reference-pool in another similar re-ordering feature for fields.
The only issue I am facing is on using clone, because on clone the node's root is the node itself rather than the RootStore which contains all the stores and thus the getRoot function doesn't resolve to the correct store.

[coolsoftwaretyler]

Ooh, nice. I forgot about that library but it sounds like a good fit. Hope it works well for you.

[k-ode]

As another suggestion, my library mst-query
has support for paginated queries and automatic normalization:

const ContactFieldGroupQuery = createQuery('ContactFieldGroupQuery', {
    data: types.model({
        fieldGroups: types.array(types.reference(ContactFieldGroupModel)),
        count: 0,
        page: 0,
        hasMore: types.boolean,
    }),
    request: types.model({ order: 'asc' }),
    pagination: types.model({ page: types.number, limit: types.number, }),
    async endpoint({ request: { order }, pagination: { page, limit } }) {
        return api.getContactFieldGroups({ page, limit, order });
    },
});

// This store will have a field "models" with all the models that the rest of your tree can reference
const ContactFieldGroupStore = createModelStore('ContactFieldGroupStore', ContactFieldGroupModel).props({
    contactFieldGroupQuery: types.optional(ContactFieldGroupQuery, {}),
}).actions(self => ({
   afterCreate() {
        onQueryMore(self.contactFieldGroupQuery, (data) => {
            self.contactFieldGroupQuery.data.fieldGroups.push(...data.fieldGroups);
        });
   }
});

const ContactFieldGroups = observer((props) => {
    const [page, setPage] = useState(0);
    const [order, setOrder] = useState('asc');
    const { data, isLoading, isFetchingMore } = useQuery(store.contactFieldGroupStore, {
        request: { order },
        pagination: { page, limit: 50 },
    });
    if (!data || isFetchingMore) {
        return <div>Loading...</div>;
    }
    return <></>
});

[aniket-banyal]

Thanks @k-ode. Will explore your library when I get time.