3.4.4. Index Module

In this chapter, you'll learn about the Index Module and how you can use it.

What is the Index Module?#

The Index Module is a tool to perform highly performant queries across modules, for example, to filter linked modules.

While modules share the same database by default, Medusa isolates modules to allow using external data sources or different database types.

So, when you retrieve data across modules using Query, Medusa aggregates the data coming from diffeent modules to create the end result. This approach limits your ability to filter data by linked modules. For example, you can't filter products (created in the Product Module) by their brand (created in the Brand Module).

The Index Module solves this problem by ingesting data into a central data store on application startup. The data store has a relational structure that enables efficiently filtering data ingested from different modules (and their data stores). So, when you retrieve data with the Index Module, you're retrieving it from the Index' data store, not the original data source.

Ingested Data Models#

Currently, only the following core data models are ingested into Index when you install it:

• Product
• ProductVariant
• Price
• SalesChannel

Consequently, you can only index custom data models if they are linked to an ingested data model. You'll learn more about this in the Ingest Custom Data Models section.

Future versions may add more data models to the list.

How to Install the Index Module#

To install the Index Module, run the following command in your Medusa project to install its package:

npm
yarn
pnpm

Then, add the Index Module to your Medusa configuration in medusa-config.ts:

medusa-config.ts
1module.exports = defineConfig({2  // ...3  modules: [4    // ...5    {6      resolve: "@medusajs/index",7    },8  ],9})

Finally, run the migrations to create the necessary tables for the Index Module in your database:

Terminal
❯npx medusa db:migrate

The index module only ingests data when you start your Medusa server. So, to ingest the currently supported data models, start the Medusa application:

npm
yarn
pnpm

The ingestion process may take a while if your product catalog is large. You'll see the following messages in the logs:

Terminal
❯info:    [Index engine] Checking for index changes❯info:    [Index engine] Found 7 index changes that are either pending or processing❯info:    [Index engine] syncing entity 'ProductVariant'❯info:    [Index engine] syncing entity 'ProductVariant' done (+38.73ms)❯info:    [Index engine] syncing entity 'Product'❯info:    [Index engine] syncing entity 'Product' done (+18.21ms)❯info:    [Index engine] syncing entity 'LinkProductVariantPriceSet'❯info:    [Index engine] syncing entity 'LinkProductVariantPriceSet' done (+33.87ms)❯info:    [Index engine] syncing entity 'Price'❯info:    [Index engine] syncing entity 'Price' done (+22.79ms)❯info:    [Index engine] syncing entity 'PriceSet'❯info:    [Index engine] syncing entity 'PriceSet' done (+10.72ms)❯info:    [Index engine] syncing entity 'LinkProductSalesChannel'❯info:    [Index engine] syncing entity 'LinkProductSalesChannel' done (+11.45ms)❯info:    [Index engine] syncing entity 'SalesChannel'❯info:    [Index engine] syncing entity 'SalesChannel' done (+7.00ms)

Enable Index Module Feature Flag#

Since the Index Module is still experimental, the /store/products and /admin/products API routes will use the Index Module to retrieve products only if the Index Module's feature flag is enabled. By enabling the feature flag, you can filter products by their linked data models in these API routes.

To enable the Index Module's feature flag, add the following line to your .env file:

Code
MEDUSA_FF_INDEX_ENGINE=true

If you send a request to the /store/products or /admin/products API routes, you'll receive the following response:

Code
1{2  "products": [3    // ...4  ],5  "count": 2,6  "estimate_count": 2,7  "offset": 0,8  "limit": 509}

Notice the estimate_count property, which is the estimated total number of products in the database. You'll learn more about it in the Pagination section.

How to Use the Index Module#

The Index Module adds a new index method to Query and it has the same API as the graph method.

For example, to filter products by a sales channel ID:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { data: products } = await query.index({13    entity: "product",14    fields: ["*", "sales_channels.*"],15    filters: {16      sales_channels: {17        id: "sc_123"18      }19    }20  })21
22  res.json({ products })23}

This will return all products that are linked to the sales channel with the ID sc_123.

The index method accepts an object with the same properties as the graph method's parameter:

• entity: The data model's name, as specified in the first parameter of the model.define method used for the data model's definition.
• fields: An array of the data model’s properties, relations, and linked data models to retrieve in the result.
• filters: An object with the filters to apply on the data model's properties, relations, and linked data models that are ingested.

How to Ingest Custom Data Models#

Aside from the core data models, you can also ingest your own custom data models into the Index Module. You can do so by defining a link between your custom data model and one of the core data models, and setting the filterable property in the link definition.

For example, assuming you have a Brand Module with a Brand data model (as explained in the Customizations), you can ingest it into the Index Module using the filterable property in its link definition to the Product data model:

src/links/product-brand.ts
1import BrandModule from "../modules/brand"2import ProductModule from "@medusajs/medusa/product"3import { defineLink } from "@medusajs/framework/utils"4
5export default defineLink(6  {7    linkable: ProductModule.linkable.product,8    isList: true,9  },10  {11    linkable: BrandModule.linkable.brand,12    filterable: ["id", "name"],13  },14)

The filterable property is an array of property names in the data model that can be filtered using the index method. When the filterable property is set, the Index Module will ingest into its data store the custom data model.

But first, you must run the migrations to sync the link, then start the Medusa application:

npm
yarn
pnpm

You'll then see the following message in the logs:

Terminal
❯info:    [Index engine] syncing entity 'LinkProductProductBrandBrand'❯info:    [Index engine] syncing entity 'LinkProductProductBrandBrand' done (+3.64ms)❯info:    [Index engine] syncing entity 'Brand'❯info:    [Index engine] syncing entity 'Brand' done (+0.99ms)

You can now filter products by their brand, and vice versa. For example:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { data: products } = await query.index({13    entity: "product",14    fields: ["*", "brand.*"],15    filters: {16      brand: {17        name: "Acme"18      }19    }20  })21
22  res.json({ products })23}

This will return all products that are linked to the brand with the name Acme. For example:

Example Response
1{2  "products": [3    {4      "id": "prod_123",5      "brand": {6        "id": "brand_123",7        "name": "Acme"8      },9      // ...10    }11  ]12}

Apply Pagination with the Index Module#

Similar to Query's graph method, the Index Module accepts a pagination object to paginate the results.

For example, to paginate the products and retrieve 10 products per page:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { 13    data: products,14    metadata15  } = await query.index({16    entity: "product",17    fields: ["*", "brand.*"],18    filters: {19      brand: {20        name: "Acme"21      }22    },23    pagination: {24      take: 10,25      skip: 0,26    }27  })28
29  res.json({ products, ...metadata })30}

The pagination object accepts the following properties:

• take: The number of items to retrieve per page.
• skip: The number of items to skip before retrieving the items.

When the pagination property is set, the index method will also return a metadata property. metadata is an object with the following properties:

• skip: The number of items skipped.
• take: The number of items retrieved.
• estimate_count: The estimated total number of items in the database matching the query. This value is retrieved from the PostgreSQL query planner rather than using a COUNT query, so it may not be accurate for smaller data sets.

For example, this is the response returned by the above API route:

Example Response
1{2  "products": [3    // ...4  ],5  "skip": 0,6  "take": 10,7  "estimate_count": 1008}

index Method Usage Examples#

The following sections show examples of how to use the index method in different scenarios.

Retrieve Linked Data Models#

Retrieve the records of a linked data model by passing in fields the data model's name suffixed with .*.

For example:

src/api/custom/products/route.ts
1const { data: products } = await query.index({2  entity: "product",3  fields: ["*", "brand.*"],4})

This will return all products with their linked brand data model.

Use Advanced Filters#

When setting filters on properties, you can use advanced filters like $ne and $gt. These are the same advanced filters accepted by the listing methods generated by the Service Factory.

For example, to only retrieve products linked to a brand:

src/api/custom/products/route.ts
1const { 2  data: products,3} = await query.index({4  entity: "product",5  fields: ["*", "brand.*"],6  filters: {7    brand: {8      id: {9        $ne: null10      }11    }12  },13})

You use the $ne operator to filter products that are linked to a brand.

Another example is to retrieve products whose brand name starts with Acme:

src/api/custom/products/route.ts
1const { 2  data: products,3} = await query.index({4  entity: "product",5  fields: ["*", "brand.*"],6  filters: {7    brand: {8      name: {9        $like: "Acme%"10      }11    }12  },13})

This will return all products whose brand name starts with Acme.

Use Request Query Configuations#

API routes using the graph method can configure default query configurations, such as which fields to retrieve, while also allowing clients to override them using query parameters.

The index method supports the same configurations. For example, if you add the request query configuration as explained in the Query documentation, you can use those configurations in the index method:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { 13    data: products,14    metadata15  } = await query.index({16    entity: "product",17    ...req.queryConfig,18    filters: {19      brand: {20        name: "Acme"21      }22    },23  })24
25  res.json({ products, ...metadata })26}

You pass the req.queryConfig object to the index method, which will contain the fields and pagination properties to use in the query.

Use Index Module in Workflows#

In a workflow's step, you can resolve query and use its index method to retrieve data using the Index Module.

For example:

src/workflows/custom-workflow.ts
1import {2  createStep,3  createWorkflow,4  StepResponse,5  WorkflowResponse,6} from "@medusajs/framework/workflows-sdk"7
8const retrieveBrandsStep = createStep(9  "retrieve-brands",10  async ({}, { container }) => {11    const query = container.resolve("query")12
13    const { data: brands } = await query.index({14      entity: "brand",15      fields: ["*", "products.*"],16      filters: {17        products: {18          id: {19            $ne: null,20          }21        }22      }23    })24
25    return new StepResponse(brands)26  }27)28
29export const retrieveBrandsWorkflow = createWorkflow(30  "retrieve-brands",31  () => {32    const retrieveBrands = retrieveBrandsStep()33
34    return new WorkflowResponse(retrieveBrands)35  }36)

This will retrieve all brands that are linked to at least one product.