Express for AI Inference in 2026: Build a Low-Latency RAG API with PGVector or FAISS

Latency stopped being optional the moment you put retrieval in front of an LLM. Every millisecond your vector search burns is a millisecond before the first token appears. Express stays lean by design, which leaves the vector store owning most of your retrieval headroom.

This guide is a project roadmap, not a speed promise. You'll wire up Express on Node.js, an embedding model, a vector store (

 or FAISS), and an LLM for answer generation. How long it takes depends on your corpus and how hard you push validation, so you get checkpoints instead of a clock.

What retrieval numbers should you plan around?

Treat published vector-search comparisons as planning baselines, not end-to-end targets. They measure retrieval. Your embedding call and LLM generation sit on top, and you need to measure those yourself.

The comparison behind these results describes FAISS as an in-memory similarity search engine optimized for local inference and pgvector as persistent, scalable vector storage. HNSW (a graph-based approximate index that trades a little exactness for speed) is a common pgvector configuration. Whatever index you choose, benchmark under realistic conditions and run warm-up queries before your service reports ready.

PGVector or FAISS: choosing by workload, not raw speed

Pick based on your workload. Pgvector lives inside Postgres, so you get persistence, SQL filters, and live updates alongside your app data. FAISS is an in-process approximate nearest neighbor library: fast, but persistence and access control become your problem.

Reach for FAISS when the corpus is fairly static and fits in memory. The same research frames it as the better fit for high-performance RAG in resource-constrained environments. Pick pgvector when documents change often or metadata filtering matters. At smaller corpora, persistence and filtering often matter more than raw speed. Either way, measure recall at the depth your re-ranker actually consumes — K=50 and K=100 when a re-ranker is downstream — not a shallow Recall@10 check that hides the gaps that hurt answer quality.

How the two stores fit into your request path

Every request in a RAG API makes five hops, and you own all of them. The client hits an Express route, the route embeds the query, the vector store returns neighbors, those chunks get packed into an LLM prompt, and the answer comes back.

client → POST /ask (Express) → embed(query) → vector search (top-K) → LLM prompt → response.

Only the vector search hop changes between the two designs. Everything else in the Express layer stays identical, which is what makes a fair head-to-head benchmark possible.

PGVector lives inside PostgreSQL. Your route makes a network call to a persistent store that also holds metadata, document versions, and access rules. FAISS runs in-process as an ANN (approximate nearest neighbor: fast, not exhaustive) index held in memory. One is a database. The other is a search library inside your service.

That split drives deployment. PGVector fits wherever Postgres already runs as its own service, so your API containers stay stateless. FAISS ties the index to each instance — every replica loads its own copy, so pair it with a companion docstore or Postgres for metadata and access rules if you enforce them at retrieval time. Because FAISS is Python- and C++-first, Node deployments need either a community wrapper or a separate process. Confirm compatibility before committing to an integration design; not every container or serverless target has been validated.

Neither, universally. Both support graph-based ANN indexing like HNSW (a layered graph that trades a little accuracy for speed), and production guides frame the real choice as trade-offs across indexing strategy, metadata filtering, incremental updates, and scaling. Performance, quality, and resource use pull against each other rather than crowning a single winner.

Choose FAISS for static corpora that fit in memory. Choose PGVector when you need live updates and per-user access filtering. Your deployment test should cover both steady-state requests and the period immediately after a new instance starts.

FAISS stops being fast the moment you reload it per request. The index lives in process memory, so load it once at boot and hand the same instance to every route. Same principle as a connection pool: one owner, many borrowers.

The pattern is a module-level singleton. It loads the index file at startup, runs synthetic warm-up queries, then flips a readiness flag. Your health check returns 503 until that flag is true, so a deploy or autoscale event never routes real users to an unprepared index. Binding details vary between Node wrappers, but this lifecycle holds across all of them.

The published numbers are from someone else's hardware

One recent evaluation clocked FAISS at 866 QPS (queries per second) on SIFT1M, and the authors note that it "lacks database operational features." The same study covered seven systems, reporting Qdrant with strong latency and Weaviate with strong out-of-the-box recall. Those results reflect the authors' hardware, datasets, and configurations. Treat them as signals, not verdicts.

That evaluation also tracks cold-start latency as its own metric, separate from steady-state percentiles. Cold and warm behavior aren't the same thing. Record them separately rather than folding startup behavior into a single headline figure.

// bench.js: pass your own search(query, k) function and real queries.
async function bench(search, queries, { warmup = 200, k = 50 } = {}) {
  const time = async (q) => {
    const t = process.hrtime.bigint();
    await search(q, k);
    return Number(process.hrtime.bigint() - t) / 1e6; // ms.
  };
  const pct = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(a.length * p)];

  const cold = [];
  for (const q of queries.slice(0, warmup)) cold.push(await time(q));

  const warm = [];
  const start = Date.now();
  for (const q of queries) warm.push(await time(q));
  const secs = (Date.now() - start) / 1000;

  console.log({
    coldP50: pct(cold, 0.5),
    warmP50: pct(warm, 0.5),
    warmP99: pct(warm, 0.99),
    qps: queries.length / secs // sequential, single-client throughput.
  });
}

The default candidate count is deliberately deeper than the final answer list. If a re-ranker sits downstream, its input size should determine the benchmark rather than an arbitrary display limit. Compare retrieval quality alongside timing — a faster configuration shouldn't get to quietly return weaker evidence.

Most of the latency you cut from a RAG API comes from work you stop doing, not clever tricks. Keep handlers async, move everything optional off the critical path, and benchmark each change against your own corpus.

Write every route as an async function. Await the embed and search calls, and pass failures to 

 so one central error handler shapes the response. Logging, analytics writes, and cache population can run after you reply. If the user doesn't need it to read the answer, it shouldn't block the answer.

import pg from 'pg';

// Pool size comes from config; validate it under load, don't copy a default.
const pool = new pg.Pool({ max: Number(process.env.PG_POOL_MAX) });

app.post('/ask', async (req, res, next) => {
  try {
    const vec = await embed(req.body.question); // your embedding client.
    const { rows } = await pool.query(
      'SELECT id, content FROM chunks ORDER BY embedding <=> $1 LIMIT $2',
      [JSON.stringify(vec), 50] // parameterized; cosine distance via pgvector
    );
    res.json(await answer(req.body.question, rows)); // your LLM call.
  } catch (err) {
    next(err); // central error handler formats the 500.
  }
});

HNSW (Hierarchical Navigable Small World, a graph-based index) needs traffic before it settles. The first queries against a cold index often run slower than steady-state, so early numbers can overstate your real latency. Send synthetic queries first, throw those results away, then record p50 and p95. Published vector database benchmarks that include pgvector with HNSW can serve as a rough sanity check at similar corpus sizes. They tell you nothing about 10 million vectors.

Every other trick is worth testing, not assuming

Shipping is where your benchmark stops being a blog post and becomes a promise. Your Express service is ready when you can show its latency, recall, and throughput on your own corpus, and keep showing them after every deploy.

Published numbers won't get you there alone. PGVector with HNSW can deliver strong throughput at moderate scale, but speed and ranking quality (measured by metrics like NDCG@100) are separate dimensions. A backend that wins on latency can still lose on retrieval quality. When you benchmark, include re-ranker depths such as K=50 and K=100, not just Recall@10. Test both on your own data before drawing conclusions.

# Implementation guidance: adjust base image and paths for your project
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY..
EXPOSE 3000
CMD ["node", "server.js"]

Real traffic makes the noise worse. Ingestion variability in vector stores is often dominated by cold-start outliers that inflate numbers well beyond steady-state behavior. Your dashboards will look the same, so separate temporary spikes from steady-state regressions before you page anyone.

Track recall next to latency and ingestion consistency. A fast index that returns the wrong chunks still fails the request. You can skip this level of rigor for a weekend prototype. Once someone else depends on your answers, you can't.

How do I integrate FAISS into a Node.js Express API when it's built for Python and C++?

FAISS has no first-class Node.js binding. A community wrapper can keep search in the application process, while a Python sidecar isolates the native dependency behind a local service boundary. The sidecar adds an operational component but can make FAISS versioning and failures easier to contain. In either design, verify the supported search operations before writing route handlers around them.

Will my FAISS-backed API hit peak throughput right after deployment?

No. Startup behavior should be treated as a separate operating phase from normal throughput. Measure it independently and keep new instances out of the request pool until their initialization work has completed. This distinction matters especially when autoscaling creates several replicas at once, because each instance has its own initialization path.

Why do my RAG answers degrade even when my vector search benchmark shows strong Recall@10?

A recall score at a shallow result depth may not represent the evidence available to the answer generator. Compare the retrieved documents against the queries that produced poor answers, then test a deeper candidate set when a re-ranker is part of the pipeline. Also inspect chunking, embedding quality, and prompt assembly; retrieval recall alone does not measure the final response.

What happens to my FAISS index when I autoscale Express replicas behind a load balancer?

Each replica loads its own copy of the FAISS index into memory, since the index is in-process rather than a shared service. A newly created instance therefore needs access to the index artifact and enough memory to hold it. PGVector takes the opposite approach: the API instances remain clients of a separate database, so the vector data is not duplicated inside every Express process.

AUTHOR

Dr. Dipen

Latency stopped being optional the moment you put retrieval in front of an LLM. Every millisecond your vector search burns is a millisecond before the first token appears. Express stays lean by design, which leaves the vector store owning most of your retrieval headroom. This guide is a project…

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

Free Tools

Express for AI Inference in 2026: Build a Low-Latency RAG API with PGVector or FAISS

Tags

Responses (0)

Free AI Career Tools

AI Job Listings

ATS Resume Checker

Startup Perks

Masterclasses

Tutorials

Fullstack React with TypeScript