SGShubham Goyal
HomeProjectsBlogContactAvailableHire me ↗

Full-stack developer building scalable systems and clean user experiences.

linkedingithubdiscordtwitterinstagramtelegram

Navigate

HomeProjectsBlogContact

Resources

RésuméGitHubLinkedIn

Get in touch

[email protected]Hire me
© 2026 SHUBHAM GOYAL · ALL RIGHTS RESERVEDPrivacy·Terms· BUILT IN INDIA · v2.0
Projects
saas·2026

Building Seekrly: Backend Architecture, Reliability, and AI Workflows

A technical deep dive into how I built Seekrly's multi-service SaaS platform: a Next.js application, Go background and AI workers, MongoDB, Redis Streams, model integrations, a Chrome extension, REST API, MCP server, and deployment. Explores real request paths, failure recovery, privacy and billing controls, trade-offs, and verified automated test coverage.

LiveExtensionAPI Docs
seekrly product image

Seekrly is a job-search CRM with applications, recruiter contacts, interviews, follow-ups, resumes, a Chrome extension, AI tools, a public REST API, and an MCP server for compatible AI clients. I built it as a multi-service product, which meant making the boundaries between the web app, background jobs, AI execution, integrations, and infrastructure explicit. This deep dive follows those boundaries, the real request paths, the failure modes I designed around, and the trade-offs that remain. It is also a record of the engineering decisions I can explain and defend—not just a list of technologies.

I will start with the system map, then follow three real user journeys before looking at persistence, queues, background processing, AI execution, integrations, deployment, and tests. The goal is to show not only what each component does, but why it exists, what can fail, and how the system recovers.

System Architecture at a Glance

text
Browser    Chrome extension    AI chat apps (MCP)    auto-apply (API key)
     |              |                  |                      |
     +--------------+------ HTTPS -----+----------------------+
                            |
                 +----------v-----------+          +---------------------+
                 |  web app (Next.js)   |          |  admin (Next.js)    |
                 |  pages, /api/v1,     |          |  internal panel     |
                 |  /api/extension, MCP |          +----------+----------+
                 |   -> service layer   |                     |
                 +---+--------------+---+                     |
                     |              | AI job (Redis stream + hash)
                     |              v                         |
                     |     +------------------+      +--------v--------+
                     |     | AI worker (Go)   |----->| Claude / OpenAI |
                     |     | 2 replicas       |      +-----------------+
                     |     +--------+---------+               |
                     v              v                         v
          +---------------------------------------------------------+
          |                  MongoDB   +   Redis                    |
          +---------------------------------------------------------+
                     ^
                     |
          +----------+--------+   HTTP   +--------------+     +------------+
          | worker (Go cron)  |--------->| Email Worker |---->| Amazon SES |
          +----------+--------+          +--------------+     +------------+
                     |
                     +--> user's own Gmail / SMTP (follow-ups)

  Files: Oracle Cloud Object Storage (private bucket, S3-compatible API)
  • Web app (Next.js 16, React 19, TypeScript): the product, the public REST API, the extension API, the MCP server and its OAuth provider, and the docs.
  • Admin panel (Next.js 16): internal tool for users, payments, AI usage, extension configs and support.
  • Background worker (Go): reminders, digests, follow-up sending, plan months, cleanup and file purging.
  • AI worker (Go): runs every AI request from a Redis queue, with model routing, personal-data redaction and credits.
  • Email Worker: an HTTP service in front of Amazon SES for Seekrly's own emails.
  • Chrome extension (Manifest V3, plain JavaScript): tracks jobs from job pages, finds contacts and autofills forms.
  • Auto-apply (Python, Playwright): searches job boards on a schedule, scores jobs and applies or bookmarks them through the public API.
  • Supporting tools: a plugin package for AI chat apps, a Scrapy and Playwright validator for extension site configs, a company-directory filler and local MCP testers.
  • Shared infrastructure: MongoDB, Redis and Oracle Cloud Object Storage, all running in Docker Compose on an Oracle Cloud VPS.
  • How the pieces interact: three journeys

    The diagram shows the parts, but the easiest way to see how they fit is to follow a few things a user actually does.

    Tracking a job from a job page

  • The extension's content script recognizes the page, either from a site config it fetched from the API or from structured data on the page, and shows a floating dock.
  • The user clicks Track Job. The background service worker calls /api/extension with a short-lived token it got through the user's web login.
  • The route hands off to the same applications service the web app uses. It checks the token's scopes, validates the input, reserves a slot in the user's monthly plan quota, writes the application to MongoDB and logs the activity.
  • The application shows up in the dashboard; the dashboard has a refresh button because the change happened outside the web app.
  • Checking job fit with AI

  • The extension asks the API to score the job. The web app builds the prompt from the job-scoring task and the user's career profile, writes it with the user's personal details to a Redis hash, and adds the job id to a Redis stream.
  • One of the AI worker replicas picks it up, redacts personal details, reserves credits, calls a model, charges the real cost, restores the redacted values and writes the answer back to Redis.
  • The web app, which has been polling the job's status, validates the answer against the task's schema and returns it to the extension. The run is also saved to the user's AI history in MongoDB, so the answer is not lost if the page that asked has closed.
  • Scheduling a follow-up email to a recruiter

  • The user writes a follow-up for a contact and picks a send time. The web app stores it in MongoDB and puts it on a Redis queue.
  • At the due time the Go worker claims it with an atomic update, decrypts the user's Gmail token or SMTP credentials, checks the sending limits shared with the app in Redis, and sends from the user's own account.
  • If the user clicks Send now instead, the app writes to a Redis stream that an always-on listener in the worker reads, so the email leaves within about a second instead of on the next minute tick.
  • The web app and its service layer

    What it does. The Next.js app serves the signed-in product, the marketing and resume-builder pages, the public REST API under /api/v1, the extension API, the MCP server and its OAuth 2.1 endpoints, and the developer docs. Every one of those entry points builds a caller and calls a function in one service layer. Services check permission scopes, validate input with zod, scope every query to the user, enforce plan limits, log writes and return a typed result that each edge turns into its own response.

    Why I built it this way. I wanted to add new ways in, first the extension, then the public API, then the MCP server, without writing new permission or quota logic each time. If scopes are checked only in route handlers, every new surface is a new chance to forget one. With checks inside the services, an MCP tool is just validated input, a service call and an output, and it inherits everything else. Keeping the API, extension API and MCP server in the same Next.js app as the product means they share that layer directly instead of calling it over the network.

    Problems I ran into. Plan limits were the trickiest part, because two requests can race. A monthly application slot is now reserved atomically before the insert and released if the insert fails. Standing totals, like the number of contacts, are checked before the insert, counted again after it and undone if the user went over. Counts are cached in Redis so this does not cost a database count on every request. Dashboard pages also made more round trips than needed; I reworked them to run independent reads in parallel, select only the fields they render and batch-fetch linked records instead of querying per row.

    Trade-offs. One app serving pages, two APIs and an MCP server is one deployable to scale together. So far that has been fine, and the slow work has been moved out to the workers anyway. The web app has no unit test suite yet; it relies on strict TypeScript, ESLint and Prettier, which is a gap I know about.

    MongoDB: one database, several writers

    What it does. MongoDB holds all persistent data. The web app, the admin panel and the Go worker all read and write it directly.

    Why. Much of the data is naturally document shaped, for example resumes are stored as JSON Resume content and rendered to PDF, and Mongoose gives the web app a single place to define the schema. Letting the worker and admin panel use the same database directly avoids building and maintaining an internal API just for them.

    Problems. The cost of several writers is that a schema change is a system-wide change. I handle it with rules rather than tooling: the client's Mongoose models are the source of truth, the admin panel mirrors them, the Go worker uses the same field names, and every change to existing documents ships with an idempotent migration script that has a dry-run mode. No code is allowed to assume a migration has run in production. A concrete example: when the user model moved from an emailVerifiedAt date to an emailVerified flag, the worker's cleanup of unverified accounts treated either field as proof of verification for a while, so a migration that had not run yet could never cause a verified account to be deleted. Once that was no longer needed I removed the extra condition.

    Other shared rules keep the writers consistent. User data is soft deleted with a deletedAt date, reads hide it automatically, and only the worker hard-deletes it 30 days later. One sessions collection holds web, admin, extension, API-key and MCP sessions, told apart by a kind field, so revoking any of them is the same operation. Dates are stored and compared in UTC, and the user's time zone is used only to compute local send times and format output.

    Trade-offs. Mirrored models in the admin panel are duplicated code that can drift, and I keep them in step by convention. In exchange there is one database to back up, monitor and reason about.

    Redis: much more than a cache

    What it does. Redis holds caches and quota counters, rate limits, the AI job queue and job data, live AI credit balances, the follow-up send queue, the Send now stream, and short-lived things like one-time codes.

    Why. The app needed Redis for rate limits and caches anyway. Redis streams with consumer groups give at-least-once delivery, a list of pending entries that another consumer can take over after a crash, and an easy dead-letter stream. That covered what I needed from a queue without running a separate message broker.

    Trade-offs. Redis is now critical: if it is down, AI requests and fast sends stop. I made sure it is never the only record of anything that matters. AI charges are also written to MongoDB, follow-ups live in MongoDB with Redis only as the queue, and a sweep job re-queues due sends that are missing from Redis.

    A queue lifecycle, from enqueue to recovery

    The important detail is not simply that a job is placed on a stream. The job payload and status live in Redis, consumers coordinate through a consumer group, and acknowledgement happens only after the outcome has been stored. If a consumer disappears before acknowledging, the entry remains pending and can be reclaimed after the configured timeout. Jobs that exhaust their attempts are moved to a dead-letter stream for inspection rather than retried forever.

    text
    Simplified lifecycle (pseudocode)
    
    Producer:   store job data -> XADD job ID to stream
    Consumer:   XREADGROUP -> load job -> process idempotently
    Success:    persist result -> XACK stream entry
    Stale job:  reclaim pending entry -> check state/attempts -> retry
    Exhausted:  write to dead-letter stream -> acknowledge original entry

    This is a simplified lifecycle, not a verbatim code listing. The key invariant is that an entry is acknowledged only after its result or terminal failure state is recorded. That narrows the failure window; idempotency and claim checks handle the cases where a job is delivered more than once.

    The background worker (Go)

    What it does. A Go service with a cron scheduler runs onboarding emails, extension and feedback nudges, inactivity reminders, an opt-in daily follow-up digest at each user's local time, interview reminders with a per-plan daily cap, scheduled follow-ups from users' own email accounts, plan-month rollovers, soft-delete cleanup, and file release and purging. An always-on listener handles Send now.

    Why a separate service in Go. Scheduled work should not depend on someone loading a page, and it should be able to run as more than one instance. Go suits a small long-running service like this: one binary in a small container, cheap concurrency, and a straightforward way to run jobs on a schedule with panic recovery and structured logging.

    The main problem: never sending an email twice. Several instances, overlapping runs, crashes and retries all create ways to double-send. Every job follows the same three rules. First, claim before you send: one atomic UpdateOne marks the work as taken, for example by setting reminderSentAt or moving nextSendAt forward, and only the instance whose update matched sends; a failed send releases the claim. Second, every system email carries an idempotency key built from what makes it unique, such as interview-reminder:<id>:<scheduledAt>, never the claim time, so a retry after a lost response is recognized downstream. Third, jobs use catch-up windows instead of exact ticks: after downtime they catch up within a bounded window and skip anything too late, so a digest more than three hours late is dropped instead of arriving in the evening.

    Billing is the one area where the worker deliberately does not own the logic. It only finds users whose plan month or paid plan has ended and calls an internal endpoint on the web app, protected by a shared secret. The web app computes plan periods from its own payments collection, where every charge, failure and refund is recorded, rather than trusting a payment provider's dates.

    Trade-offs. Claims and idempotency keys add a field or two and some care to every job. In exchange, scaling the worker is just running another container.

    Email: Amazon SES for us, the user's own account for outreach

    There are two kinds of email, and they take different paths on purpose.

    Seekrly's own emails (verification codes, reminders, digests) go through a separate Email Worker HTTP API in front of Amazon SES. The web app, admin panel and worker never speak SMTP. POST /emails validates the message, queues it in a Redis stream and returns 202. A worker pool checks a global blacklist, waits for a slot in a global rate limit and calls SES. Transactional mail has priority, but general mail always gets a share of reads so it is never starved. General mail is retried with backoff on transient errors; errors a retry cannot fix go to a dead-letter queue. Every state change is kept in MongoDB, and SES delivery, bounce and complaint notifications come back through an SNS webhook and feed suppression. Callers pass a stable job id, which is what makes the worker's idempotency keys work end to end.

    Why. SES is cheap and reliable for transactional email, and putting one service in front of it means rate limits, suppression and history are handled once for every project instead of in each one.

    Outreach to recruiters is different: it goes from the user's own Gmail or SMTP account, so replies land in their inbox and the email comes from them. Gmail is connected with permission to send only, not to read mail, and refresh tokens and SMTP passwords are encrypted at rest with AES-256-GCM. Sending limits per user, per minute and per recipient are shared between the app and the worker through Redis, so a bug or an over-eager user cannot spam anyone.

    The AI worker (Go)

    What it does. Every AI feature, from job scoring and screening answers to resume tailoring, cover letters and interview prep, goes through this worker. In the app, a feature is just a task: a prompt, input and output zod schemas, and a tier. Registering a task makes it available in the dashboard, the extension and the public API. The worker never knows which feature it is running.

  • Load the job from Redis and turn attachments such as PDF and DOCX resumes into text, so a raw file never reaches a provider.
  • Check size and look for prompt injection in third-party text such as job posts; if found, tell the model to treat that text as data.
  • Replace personal details with placeholders like [[NAME_1]] and [[EMAIL_1]]: the user's own details, plus any other emails, phone numbers and profile links the detectors find.
  • Reserve the worst-case cost in credits with one atomic Lua script against a Redis balance.
  • Route the tier (fast, balanced or smart) to an ordered list of Claude and OpenAI models, retrying rate limits with backoff and falling back on refusals or invalid JSON, with answers constrained by structured output.
  • Charge the real cost from token usage and record every attempt in a ledger in MongoDB.
  • Remove contact details the model invented, then put the real values back in place of the placeholders.
  • Write the answer back, delete the inputs from Redis, and acknowledge the job only after the outcome is stored.
  • Why one worker. Model calls are slow, fail in many different ways and cost real money. Doing redaction, fallbacks and billing inside each feature would mean doing them many times, slightly differently. With one worker, privacy and billing are solved once, and a new AI feature is a prompt and a schema in the app. Go also gives the worker the official Anthropic SDK, and OpenAI or any compatible endpoint plugs in through one generic agent.

    Credits. Cost is computed from real token usage, including cached input and cache writes:

    text
    cost (micro-USD) = uncached input x input price + cached input x cached price
                     + cache writes x cache-write price + output x output price
    credits          = ceil(cost x markup / microUsdPerCredit), at least 1 per call

    The live balance sits in Redis so no job has to touch MongoDB to check credits. The reservation assumes the most expensive candidate model, no cache hits and the full output budget, and is settled to the real cost afterwards. The MongoDB ledger is the record: a missing Redis balance is rebuilt from it, and once a day one worker, holding a Redis lock, recounts every balance from the ledger and corrects any drift. Users only ever see the percentage of credits left.

    Problems. Crashes in the middle of a job were the main risk: a reserved hold could be left behind, or a job could run twice. A job not finished within a reclaim timeout is taken over by another replica, and the hold it left is refunded first. The worker refuses to start if that timeout is not longer than the job timeout, so a live job is never picked up twice. Jobs that use up their attempts go to a dead-letter stream. Debugging prompts was another problem, since nothing personal is ever logged. I added a temporary trace stored on each AI run in MongoDB and shown only in the admin panel.

    Trade-offs. Any attempt the model answered is charged, even an unusable answer or a refusal, so prompts designed to make the model fail cannot spend for free; provider errors are never charged, and an admin can refund a run by hand. The temporary trace keeps personal details next to the prompt, which is why it is admin-only and meant to be switched off once it is no longer needed. Model choices and prices can change through Redis without a deploy, but provider endpoints and keys only come from deploy config, so nobody can redirect prompts by editing Redis.

    File storage

    What it does. Resumes and photos are rows in a files collection with a reference count and the name of the store that holds the bytes. The app adds and removes references as resumes and profiles change; the worker releases files whose count reaches zero and purges their bytes 30 days later.

    Why. With a reference count, nothing ever scans other collections to find out whether a file is still used. Recording the store on each row let me change storage without migrating old files: new uploads go to Oracle Cloud Object Storage through its S3-compatible API, with Vercel Blob and MongoDB as fallbacks in that order, while older files are still read and purged from whichever store holds them. The bucket is private, and every read is streamed through the app's own authenticated routes.

    Problems. Moving to Oracle Cloud Object Storage hit two snags: the S3-compatible endpoint needed the correct customer-oci.com domain, and the not-found check had to tell a genuinely missing object apart from a configuration error, so a misconfigured bucket is never treated as a missing file.

    The Chrome extension

    What it does. A floating dock on job pages (Track Job, Job fit, Auto-fill, Contact) and a side panel with the current job, weekly goals, applications, contacts and follow-ups. It recognizes jobs from per-site configs or, on any other site, from JSON-LD, microdata, ATS URLs and page structure, and it autofills application forms from the user's profile.

    How it talks to the rest. It never holds a long-lived API key. It signs in through the user's web login with a handshake on the Seekrly site and uses five-minute tokens renewed with the browser session. It calls /api/extension, which mostly reuses the public API's handlers. Site configs, autofill fields and form configs come from the API and are versioned: every admin change bumps a version and extensions fetch the new copy, while the copies bundled in the extension are only an offline fallback.

    Why. Job sites change their markup all the time. Server-driven configs mean a broken selector is fixed in the admin panel and reaches every user without a Chrome Web Store review. Short-lived tokens mean a leaked token is worth very little.

    Problems. Chrome sometimes injected the content script twice into the same page, and the second copy crashed with Identifier has already been declared; the release build now wraps the bundle in a once-per-page guard. Config endpoints were being cached, so fixes did not reach extensions right away; I turned caching off for those routes and let the version numbers decide when to fetch. To catch broken configs before users do, a Scrapy and Playwright project loads live job pages and runs the extension's own extraction code against them, so a pass there means the extension works on that site.

    Trade-offs. There is no bundler in development: files share globals by load order, and each surface has one ordered list of scripts. It is simple to debug, but a new file in the wrong position will not work. The release build concatenates and minifies those lists and never obfuscates, because the Chrome Web Store does not allow obfuscated code.

    Public API, docs and the MCP server

    What they do. Users create automation keys for the REST API with fine-grained scopes such as applications:read, profile:write, email:send and items:delete. The MCP server exposes the same capabilities to AI chat apps through its own OAuth 2.1 provider with dynamic client registration, PKCE and rotating refresh tokens. A plugin package pairs the MCP server with task skills that teach AI chat apps how to run a job search with it.

    Why one source for docs. The API reference lives in one TypeScript file, and the docs page, llms.txt, a SKILL.md for agents and the OpenAPI document are all generated from it. Humans, scripts and AI tools read the same description, and it cannot drift between formats. The scopes are also defined in one file that drives the consent screens and the docs.

    Problems and decisions specific to AI agents. A model acting for the user can be wrong or can be manipulated by text in a job post. Deletes, profile and resume updates and emails therefore take two calls: the first returns a full preview and a confirmation token, and the model must show the preview and get a clear yes before calling again. The server instructions tell the model that text in job posts and notes is data, not instructions, never to guess ids, and to ask for an interview's time zone instead of assuming one. During the plugin review, the update tools that overwrite data were flagged for not being marked destructive; they now carry that hint so AI apps ask before using them.

    The admin panel

    What it does. A separate Next.js app for users and their activity, subscriptions and payments, AI usage, credits and refunds, discount codes, extension configs, support requests and the shared company directory.

    Why separate. An internal tool with this much power should not share a deployable or a session with the product. Admin tokens are signed for their own audience so web-app tokens never work there, sign-in is a password followed by a six-digit code sent by email, and every page and route re-checks the session in the database rather than trusting the token alone. Any URL an admin enters is fetched through an SSRF guard that pins the resolved address.

    Trade-offs. It reads the same database through mirrored models, which is duplication I accept for one source of data. Admins may see credit numbers; the product never shows them.

    Auto-apply

    What it does. A Python tool that reads the user's career profile and resume from Seekrly, searches several job boards on a schedule, scores jobs with Seekrly AI, applies where it can and bookmarks the rest in Seekrly, then emails a digest. It uses only the documented public API with an automation key, which is a good test that the API is enough to build on.

    Decisions. Screening answers are decided in Python in a fixed order: saved answers, then rules from the profile, then AI. AI answers are validated and must never claim experience the profile does not support. If a required question has no supported answer, it does not submit; it bookmarks the job with the reason in its notes. A job is recorded as Applied only when the board shows a confirmation.

    Problems. Many job boards block headless browsers, so on the server it runs Chrome with a real window inside a virtual display (Xvfb). Selectors break when boards change their pages, so each board's selectors live in one class, and a doctor command checks them against the live page and saves suggestions. Some boards' internal APIs drift too, for example returning empty job details, so it prefers fields from search results.

    The company directory

    Companies are the only data that can become public, and only after an admin approves them. A Python tool fills the shared directory from Wikidata and companies' own homepages and keeps the source on every record. A wrong match is worse than none, so it takes a Wikidata entry only when its website matches or the name is unique, even though that lowers the hit rate. In the app, duplicate-company suggestions used to match unrelated companies, because empty names matched each other and companies created from the same job board shared its domain. Empty values no longer match, and a domain shared by more than two of a user's companies is ignored.

    Deployment

    Everything runs on an Oracle Cloud VPS. Each service has its own Dockerfile and Docker Compose file and joins shared external Docker networks for the reverse proxy, the database and the email service, so services only see the networks they need. The web app is built in a multi-stage Node 24 Alpine image. The AI worker runs two replicas by default and scales with docker compose --scale; on shutdown it stops reading and lets running jobs finish.

    yaml
    services:
       worker:
          image: seekrly-ai-worker:latest
          restart: unless-stopped
          # Jobs are shared through one Redis consumer group. Override for a
          # while with `docker compose up -d --scale worker=N`.
          deploy:
             replicas: ${WORKER_REPLICAS:-2}
          stop_grace_period: 3m

    Trade-offs. A single VPS with Docker Compose is cheap and gives me full control, but it is one machine and I am the operations team. Keeping the workers stateless and every job idempotent is what makes it reasonable to scale out or move later.

    Testing

    The strongest automated coverage is around the services where retries, scheduling, and external failures are most consequential: 84 Go test functions for the background worker, 59 Go test functions for the AI worker, and about 170 offline pytest tests for auto-apply. These are test-function counts, not a claim of overall code coverage. The web app and admin panel do not yet have a unit test suite, which remains an explicit gap.

  • Background worker: 84 Go test functions, including integration tests against a local MongoDB for which users get picked, time zones, claims and the daily reminder cap. A command renders every email template without the email API or a database.
  • AI worker: 59 Go test functions that run offline with an in-memory Redis, fake models, fake billing and local HTTP test servers, plus credit tests against a real MongoDB. A mock mode returns schema-shaped answers with random delays, failures and refusals, so the whole flow can be tried without provider keys.
  • Auto-apply: about 170 offline pytest tests that never open a browser or touch the network.
  • Web app and admin panel: strict TypeScript, ESLint and Prettier, but no unit tests yet. Extension configs are checked against live pages, and the plugin package has its own validation script.
  • What I would tell myself at the start

  • Put permission checks next to the data. One service layer made every later surface, including AI agents, cheap and safe to add.
  • Assume everything will be retried. Atomic claims and idempotency keys cost little on day one and are painful to add later.
  • Keep risky integrations behind one door. AI providers, email and file storage each sit behind one service or interface, so privacy, billing, limits and migrations happen in one place.
  • Never let a cache be the only record. Redis makes things fast, but MongoDB always holds what is needed to rebuild it.
  • Category
    saas
    Year
    2026
    Links
    LiveExtensionAPI Docs

    Have a project in mind?

    Start a project ↗
    All projects