server.js, worker.js, and zsh A Practical Course

server.js, worker.js, and zsh — A Practical Course --- Part 0: The Mental Model Before any code, understand the shape of things.

server.js, worker.js, and zsh — A Practical Course

---

Part 0: The Mental Model

Before any code, understand the shape of things.

A typical backend has **two kinds of processes**:

| | `server.js` | `worker.js` |
|---|---|---|
| **Job** | Answer requests fast | Do slow work in the background |
| **Triggered by** | HTTP request from a user | A message on a queue |
| **Must respond in** | ~100ms | Minutes is fine |
| **Example** | "Save this document" → returns `202 Accepted` | Chunk the doc, embed it, store vectors |
| **If it crashes** | Users see errors immediately | Job retries, users unaffected |

The rule: **anything slow, expensive, or flaky does not belong in `server.js`.**

Why? An HTTP request holds a connection open. If embedding a 200-page contract takes 40 seconds, the browser times out, the load balancer kills it, and the user has no idea if it worked. Instead, `server.js` accepts the work, writes a row saying "job pending," and returns instantly. `worker.js` picks up the job whenever it's ready.

This split is why your legal document engine needs both. Upload a PDF → `server.js` stores the file and says "processing." `worker.js` extracts text, chunks it, embeds each chunk, writes to pgvector. User polls or gets notified.

---

Part 1: `server.js`

1.1 — The minimum viable server

```js
// server.js
import express from "express";

const app = express();
app.use(express.json());

app.get("/health", (req, res) => {
res.json({ ok: true });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
console.log(`listening on ${PORT}`);
});
```

Three things happening:

1. **`express()`** creates an app object — a container for routes.
2. **`app.use(express.json())`** is *middleware*. It runs on every request before your route handler. This one parses JSON request bodies into `req.body`.
3. **`app.listen()`** binds to a TCP port and starts accepting connections.

1.2 — Anatomy of a request handler

```js
app.post("/documents", async (req, res) => {
const { title, content } = req.body;
// ↑ input from the client — never trust it

if (!title) {
return res.status(400).json({ error: "title required" });
}

const doc = await db.documents.insert({ title, content });

res.status(201).json({ id: doc.id });
});
```

`req` — everything the client sent (body, headers, query params, cookies)
`res` — how you reply (status code, headers, body)
`async` — because database calls take time; `await` pauses without blocking other requests

**Critical:** `return` before `res.status(400)`. Without it, execution continues and you'll try to send two responses. Node will throw `ERR_HTTP_HEADERS_SENT`.

1.3 — Middleware, properly understood

Middleware is a function with the signature `(req, res, next)`. It sits between the incoming request and your handler.

```js
function requireAuth(req, res, next) {
const token = req.headers.authorization?.replace("Bearer ", "");

if (!token) {
return res.status(401).json({ error: "unauthorized" });
}

const user = verifyToken(token); // throws if invalid
req.user = user; // attach for downstream handlers

next(); // ← hand control to the next thing in the chain
}

app.post("/documents", requireAuth, async (req, res) => {
// req.user is guaranteed to exist here
const doc = await db.documents.insert({
title: req.body.title,
firm_id: req.user.firm_id, // ← scoping data to the user's org
});
res.status(201).json({ id: doc.id });
});
```

That `firm_id` line is the whole ballgame for multi-tenant RAG. If you forget it in *one* query, Firm A retrieves Firm B's privileged documents. This is the cross-user retrieval leak. It's a query-scoping bug, not an AI bug.

1.4 — Error handling

Async errors don't propagate to Express automatically in older versions. Wrap them:

```js
const asyncHandler = (fn) => (req, res, next) =>
Promise.resolve(fn(req, res, next)).catch(next);

app.post("/documents", requireAuth, asyncHandler(async (req, res) => {
const doc = await db.documents.insert({ /* ... */ });
res.status(201).json({ id: doc.id });
}));

// The error handler — 4 args tells Express this is the error middleware
app.use((err, req, res, next) => {
console.error(err);
res.status(500).json({ error: "internal error" });
// ↑ never leak err.message to clients
});
```

Never send `err.message` to the client. Stack traces leak table names, file paths, and library versions.

1.5 — Graceful shutdown

Railway, Fly, Kubernetes — they all send `SIGTERM` before killing your process. If you ignore it, in-flight requests get dropped.

```js
const server = app.listen(PORT);

process.on("SIGTERM", () => {
console.log("SIGTERM received, draining...");
server.close(() => { // stop accepting new connections
db.end(); // close DB pool
process.exit(0);
});

// Force exit if drain takes too long
setTimeout(() => process.exit(1), 10_000).unref();
});
```

1.6 — The document upload endpoint, for real

```js
app.post("/documents", requireAuth, asyncHandler(async (req, res) => {
const { title, storage_path } = req.body;

// 1. Record the job. Status is the source of truth.
const doc = await db.documents.insert({
title,
storage_path,
firm_id: req.user.firm_id,
status: "pending",
});

// 2. Enqueue the slow work
await queue.add("ingest-document", { document_id: doc.id });

// 3. Return immediately. 202 = "accepted, not finished"
res.status(202).json({ id: doc.id, status: "pending" });
}));
```

Note what this handler does *not* do: no PDF parsing, no embedding, no vector writes. It writes one row and pushes one message. Total time: maybe 20ms.

---

Part 2: `worker.js`

2.1 — What a worker actually is

A worker is a long-running Node process with **no HTTP server**. It connects to a queue, pulls jobs, and processes them in a loop. It runs as a separate deployment from `server.js` — separate container, separate scaling, separate crash domain.

```js
// worker.js
import { Worker } from "bullmq";

worker.on("completed", (job) => console.log(`done: ${job.id}`));
worker.on("failed", (job, err) => console.error(`failed: ${job.id}`, err));
```

`concurrency: 3` matters. If each job calls an embedding API with a rate limit, unbounded concurrency will get you 429s. Tune this to your slowest downstream dependency.

2.2 — The ingestion pipeline

```js
async function ingestDocument(documentId) {
const doc = await db.documents.findById(documentId);
await db.documents.update(documentId, { status: "processing" });

try {
// 1. Fetch the file
const buffer = await storage.download(doc.storage_path);

// 2. Extract text
const text = await extractText(buffer);

// 3. Chunk it
const chunks = chunkText(text, { size: 1000, overlap: 200 });

// 4. Embed — batched, not one-by-one
const embeddings = await embedBatch(chunks.map((c) => c.text));

// 5. Write to pgvector, scoped to the firm
await db.chunks.insertMany(
chunks.map((chunk, i) => ({
document_id: documentId,
firm_id: doc.firm_id, // ← carried through every layer
content: chunk.text,
page: chunk.page, // ← for citations
embedding: embeddings[i],
}))
);

await db.documents.update(documentId, { status: "ready" });
} catch (err) {
await db.documents.update(documentId, {
status: "failed",
error: err.message,
});
throw err; // ← rethrow so the queue knows to retry
}
}
```

Two things worth staring at:

**`throw err` at the end.** If you swallow the error, BullMQ marks the job successful and never retries. Always rethrow.

**`page: chunk.page`.** Store the page number *at chunk time*. Legal citations need "Contract §4.2, page 12." If you don't capture provenance during ingestion, you cannot reconstruct it at query time.

2.3 — Idempotency

Queues guarantee *at-least-once* delivery, not exactly-once. A worker can crash after writing chunks but before marking the job complete. The job retries. Now you have duplicate chunks, and every retrieval returns the same passage three times.

Make the write idempotent:

```js
// Delete existing chunks before inserting. Safe to run 100 times.
await db.chunks.deleteWhere({ document_id: documentId });
await db.chunks.insertMany(newChunks);
```

Or use a unique constraint on `(document_id, chunk_index)` and `ON CONFLICT DO NOTHING`.

2.4 — Retries and backoff

```js
await queue.add(
"ingest-document",
{ document_id: doc.id },
{
attempts: 3,
backoff: { type: "exponential", delay: 5000 }, // 5s, 10s, 20s
removeOnComplete: 100, // keep last 100 for debugging
removeOnFail: 1000,
}
);
```

Exponential backoff exists because failures are often transient — a rate limit, a network blip, a database failover. Retrying instantly just hammers the thing that's already struggling.

**Distinguish retryable from permanent failures.** A 429 should retry. A corrupt PDF should not — it will fail identically three times and burn your queue capacity.

```js
class PermanentError extends Error {}

// In the worker:
catch (err) {
if (err instanceof PermanentError) {
await job.discard(); // don't retry
}
throw err;
}
```

2.5 — Worker graceful shutdown

Same principle as the server, different mechanics. You want the current job to *finish*, not get killed halfway through writing vectors.

```js
process.on("SIGTERM", async () => {
await worker.close(); // stops taking new jobs, waits for active ones
await db.end();
process.exit(0);
});
```

2.6 — Sharing code between server and worker

Both processes need the DB client, the schema, the config. Structure it so neither imports the other:

```
src/
server.js ← entrypoint 1
worker.js ← entrypoint 2
lib/
db.js
queue.js
config.js
jobs/
ingest-document.js ← the pure function worker.js calls
```

`server.js` imports `lib/queue.js` to *enqueue*. `worker.js` imports it to *consume*. Neither knows the other exists.

---

Part 3: zsh

3.1 — Why zsh and not bash

zsh is the default shell on macOS since Catalina. It's bash-compatible for practical purposes but has better tab completion, better globbing, and a real plugin ecosystem. Everything in Part 3 assumes zsh; most of it works in bash too.

3.2 — The startup files

This confuses everyone. There are two files you care about:

| File | When it runs |
|---|---|
| `~/.zshenv` | **Every** zsh invocation, including scripts |
| `~/.zshrc` | Only for **interactive** shells |

Practical rule:
- `PATH` and environment variables that scripts need → `~/.zshenv`
- Aliases, prompt, plugins, anything you only need when typing → `~/.zshrc`

If you put `PATH` in `.zshrc`, a cron job or a script run with `zsh script.sh` won't see it. This is the source of about 80% of "works in my terminal, fails in CI" bugs.

3.3 — Environment variables and the `$` sigil

```zsh
export DATABASE_URL="postgres://localhost:5432/mydb"
# ↑ export makes it visible to child processes

echo $DATABASE_URL # read it
echo "${DATABASE_URL}" # braces disambiguate: "${VAR}_suffix"
```

Without `export`, the variable exists in your shell but `node server.js` won't see it. This is why `.env` files exist — they're a workaround for not wanting to export a dozen things.

```zsh
# Load a .env file into your current shell
set -a # auto-export everything defined from here
source .env
set +a # stop auto-exporting
```

3.4 — Quoting, which is where everyone gets burned

```zsh
name="Khurram Ali"

echo $name # → Khurram Ali (word-split into 2 args, usually fine)
echo "$name" # → Khurram Ali (one arg — what you want)
echo '$name' # → $name (single quotes: no expansion at all)
```

**Always double-quote your variables.** Consider:

```zsh
file="my document.pdf"
rm $file # ← tries to delete "my" AND "document.pdf"
rm "$file" # ← correct
```

Single quotes are literal. Double quotes expand `$vars` and `$(commands)` but not globs. Backslash escapes one character.

3.5 — Redirection and pipes

```zsh
node server.js > out.log # stdout → file (overwrite)
node server.js >> out.log # stdout → file (append)
node server.js 2> err.log # stderr → file
node server.js > all.log 2>&1 # both → same file
node server.js 2>&1 | grep ERROR # both → pipe
node server.js > /dev/null 2>&1 # discard everything
```

The `2>&1` means "make file descriptor 2 (stderr) point where fd 1 (stdout) points." **Order matters.** `> file 2>&1` works. `2>&1 > file` does not — it points stderr at the old stdout (the terminal), *then* moves stdout to the file.

3.6 — Command substitution and pipelines

```zsh
COMMIT=$(git rev-parse --short HEAD)
echo "Deploying $COMMIT"

Chain with && (only if previous succeeded) and || (only if it failed)

Find and kill whatever's on port 3000

`xargs` takes stdin and turns it into command arguments. `kill -9` doesn't read stdin, so you need `xargs` to bridge them.

3.7 — Scripting: the safety preamble

Every script you write should start with this:

```zsh
#!/usr/bin/env zsh
set -euo pipefail
```

`set -e` — exit immediately if any command fails
`set -u` — error on undefined variables (catches typos)
`set -o pipefail` — a pipeline fails if *any* stage fails, not just the last

Without `pipefail`, `false | true` exits 0. Your deploy script "succeeds" while the build silently failed.

3.8 — A real deploy script

```zsh
#!/usr/bin/env zsh
set -euo pipefail

: "${RAILWAY_TOKEN:?RAILWAY_TOKEN must be set}"
# ↑ fail loudly with a message if unset

log() { print -P "%F{cyan}[$(date +%H:%M:%S)]%f $*"; }
# ↑ zsh prompt escapes: %F{color}...%f

log "Running tests"
npm test

log "Building"
npm run build

log "Deploying server"
railway up --service server

log "Deploying worker"
railway up --service worker

log "Done ✓"
```

The `:?` syntax is the cleanest way to assert an env var exists. It fails at line 4 with a clear message instead of at line 40 with `command not found`.

3.9 — Functions and loops

```zsh
# Function
retry() {
local attempts=$1; shift
local i=1
until "$@"; do
if (( i >= attempts )); then
print "failed after $attempts attempts" >&2
return 1
fi
print "attempt $i failed, retrying..." >&2
sleep $(( 2 ** i )) # exponential backoff
(( i++ ))
done
}

retry 3 curl -f https://api.example.com/health
```

`local` scopes the variable to the function
`shift` drops `$1`, so `"$@"` becomes the command to run
`(( ))` is arithmetic context — no `$` needed on variables inside

```zsh
# Loop over files
for f in *.pdf; do
print "processing $f"
node scripts/ingest.js "$f"
done

Loop over lines in a file

`IFS= read -r` is the correct incantation. `IFS=` prevents trimming whitespace; `-r` prevents backslash mangling.

3.10 — Aliases and functions for daily work

Put these in `~/.zshrc`:

```zsh
# Aliases — simple text substitution
alias gs="git status -sb"
alias gd="git diff"
alias ll="ls -lah"
alias dc="docker compose"

Functions — when you need arguments

Kill whatever's squatting on a port

Tail logs from your Railway service

`${1:-server}` means "use `$1`, or `server` if `$1` is empty."

3.11 — Running both processes locally

```zsh
# Terminal 1
npm run dev:server

Terminal 2

Or use a process manager. In `package.json`:

```json
{
"scripts": {
"dev": "concurrently -n server,worker -c cyan,magenta \"npm:dev:*\"",
"dev:server": "node --watch src/server.js",
"dev:worker": "node --watch src/worker.js"
}
}
```

`node --watch` is built in since Node 18 — no nodemon needed.

---

Part 4: How It Fits Together

The request flow through your legal document engine:

```
POST /documents
│
▼
┌──────────────┐ 1. insert row (status: pending)
│ server.js │ 2. queue.add("ingest", { id })
│ │ 3. return 202 { id, status }
└──────┬───────┘
│ ~20ms, connection closed
▼
┌───────┐
│ Redis │ ← job sits here until a worker picks it up
└───┬───┘
│
▼
┌──────────────┐ 1. download PDF
│ worker.js │ 2. extract text
│ │ 3. chunk (with page numbers)
│ │ 4. embed (batched)
│ │ 5. insert into pgvector (with firm_id)
│ │ 6. update status: ready
└──────────────┘
│ ~40s, nobody waiting
▼
GET /documents/:id → { status: "ready" }
```

And the query flow:

```
POST /search { query: "termination clause" }
│
▼
┌──────────────┐ 1. embed the query
│ server.js │ 2. SELECT ... WHERE firm_id = $1
│ │ ORDER BY embedding <=> $2 LIMIT 10
│ │ 3. return chunks + page citations
└──────────────┘
```

Search is fast enough to live in `server.js` — one embedding call plus one indexed query, maybe 200ms. Ingestion is not. That's the entire dividing line.

---

Part 5: Exercises

1. **Server.** Add a `GET /documents/:id` endpoint that returns the document's status. Scope it to `req.user.firm_id` and return `404` (not `403`) if it belongs to another firm. *Why 404?* Because `403` confirms the document exists.

2. **Worker.** Make `ingestDocument` idempotent. Run it twice on the same document and verify the chunk count doesn't double.

3. **Worker.** Add a `PermanentError` for PDFs with zero extractable text (scanned images). Verify the job doesn't retry.

4. **zsh.** Write `scripts/reset-db.sh` that: refuses to run if `$NODE_ENV` is `production`, drops and recreates the local DB, and runs migrations. Use `set -euo pipefail` and the `:?` guard.

5. **zsh.** Write a function `waitfor <url>` that polls until the URL returns 200 or 30 seconds pass. Use it in your dev script to wait for the server before starting the worker.

6. **Both.** Trace what happens if the worker crashes between step 5 (insert vectors) and step 6 (update status). What does the user see? How do you fix it?

---

Reference Card

**server.js**
- Middleware runs in order; `next()` passes control
- Always `return` before `res.send()` in guard clauses
- Scope every query by tenant ID — no exceptions
- Slow work → queue, return `202`
- Handle `SIGTERM`, drain connections

**worker.js**
- No HTTP server, separate deployment
- Rethrow errors or the queue thinks you succeeded
- Assume at-least-once delivery → make writes idempotent
- Bound concurrency to your slowest dependency
- Distinguish retryable from permanent failures
- Capture provenance (page numbers) at ingest time

**zsh**
- `~/.zshenv` for PATH, `~/.zshrc` for interactive
- `set -euo pipefail` at the top of every script
- Always `"$quote"` your variables
- `${VAR:?message}` to assert required env vars
- `2>&1` after the redirect, not before
- `(( ))` for math, `[[ ]]` for tests