Esc Navindexer

Esc Navindexer is a Shopify app which uses a system of data collection and processing to provide advanced collection filtering and display options on Shopify stores.

Esc Navindexer is hosted on a fault-tolerant AWS infrastructure.

Services overview

The system is comprised of three major services:

• Frontend instances
  Provides the Slice API and App Area.
• A single Scheduler instance
  Determines when indexing needs to take place and pushes jobs into the queue.
• Worker instances
  Processes indexing jobs created by the schduler.

Scheduler

The scheduler's only job is to determine when full reindexes need to take place and to push jobs onto the queue. The Scheduler is currently a single-point-of-failure in the system. A future update is intended to allow multiple schedulers to run simulanteously, removing the SPOF.

The current scheduler can be accessed at https://scheduler.navindexer.eastsideco.io/

If the current scheduler begins to fail, it will be terminated and replaced automatically by autoscaling.

Workers

The worker instances take index jobs from the queue and process them. For a full reindex, they must gather information on every product, variant, and collection on the store.

Workers can be accessed at https://workers.navindexer.eastsideco.io/ (this will choose a worker from the group based on load).

Workers can currently scale between 1-3 instances based on CPU load. This may need to be tested, as the workers can become saturated with necessarily hitting high CPU.

Frontends

The frontend instances provide the App Area and Slice API.

The App Area is the dashboard and configuration area accessible to merchants via the EASDK. It can be accessed at https://navindexer.eastsideco.io/app (this will choose a frontend based on load).

The Slice API is used on shop collection pages to retrieve collection, product, and rendering information. Is can be accessed at https://navindexer.eastsideco.io/api/v1/slice (this will choose a frontend based on load).

Frontends can currently scale between 1-3 instances based on CPU load.

Architecture

The Esc Navindexer architecture consists of:

• 3 classic Elastic Load Balancers (frontends, workers, schedulers)
• 2 EC2 Launch Configurations (workers, non-workers)
  The LCs provide the type of instance to the application via User Data.
• 3 EC2 Auto-Scaling Groups (frontends, workers, schedulers)
  The ASGs recieve traffic from the three ELBS
• 1 RDS Multi-AZ MySQL DB
• 1 S3 bucket with Code Deploy Application Revisions
• 3 Code Deploy groups for deploying Revisions (frontends, workers, schedulers)

At the moment, all resources are available only in eu-west-1a. A high-availability configuration is intended as a future option.