Job Listing Fullstack Dashboard + API
A FastAPI and React dashboard over the scraped job corpus. The API serves the corpus read-only with a per-user layer for saved jobs, an admin operations page queues scraper runs for a worker process that reports progress back through the database, and an optional AI layer answers questions over the same filtered data.
- Role
- Sole developer. API, worker, data synchronisation, object store, AI layer, frontend.
- Context
- Personal project built on the Indonesia Job Data Scraping System (case 001). The API runs on Vercel, the worker on the machine that has the scraper.
- Period
- 2026-09 → ongoing (Repository created 2026-09-04.)
- Evidence
- ArchitectureDocumentationTechnical breakdown
01
The Case
The scraping pipeline (case 001) produces a growing corpus of job posts as JSON, CSV, and SQLite on one machine. Reading it meant opening files. The goal was a dashboard that lets a user browse, filter, and save vacancies, lets an admin start and watch scraper runs from a browser, and can be deployed on a platform that has no access to that machine.
SCRAPING -> PROCESSING -> API -> DASHBOARD -> USER
(case 001) (worker sync) (FastAPI on Vercel) (React)
02
Evidence
Repository README: architecture, operations, AI features, deployment
document
Backend README: databases, corpus sync, image proxy, endpoints
document
03
Architecture
Text version of the diagram
Data source
-
Scraping pipeline (case 001)
external
Writes the SQLite corpus and image cache on the worker machine.
API and storage
-
FastAPI on Vercel
FastAPI
Auth, jobs, stats, analytics, saved jobs, operations queue, admin, AI endpoints.
-
Neon Postgres
PostgreSQL
Users, saved jobs, ops queue and logs, worker heartbeats, AI notes, and the synced corpus copy.
Worker machine
-
Worker process
Python
Polls the queue, runs scraper, account search, clean, and autofill jobs as subprocesses, heartbeats every 10 s, syncs SQLite to Postgres.
-
SQLite corpus
SQLite
Opened read-only (mode=ro) by the API and the worker.
-
Image bucket
S3-compatible object storage
Worker uploads cached flyer images; API redirects with presigned URLs.
Client
-
React dashboard
React
Vite + React 19 + Recharts; KPI overview, filterable list, job page, saved jobs, admin operations.
- User / admin external
AI providers
-
Groq / Anthropic
Groq APIexternal
Tool-calling agent over the same filter layer; mock provider in tests.
Connections
- User / admin → React dashboard · browser
- React dashboard → FastAPI on Vercel · JSON (HTTPS)
- FastAPI on Vercel → Neon Postgres · SQLAlchemy
- FastAPI on Vercel → Groq / Anthropic · chat / enrich (HTTPS)
- Worker process ↔ Neon Postgres · claim jobs, logs, heartbeat
- Worker process → Scraping pipeline (case 001) · subprocess
- Scraping pipeline (case 001) → SQLite corpus
- SQLite corpus → Worker process · sync to Postgres
- Worker process → Image bucket · upload images
- FastAPI on Vercel → Image bucket · presigned redirect
04
Technical Breakdown
application
FastAPI with SQLAlchemy 2.0, JWT authentication (PyJWT, bcrypt), roles admin and user.
Endpoint groups: auth, jobs, stats, analytics, meta, runs, saved, ops, admin, and ai.
The worker claims queued rows with a conditional UPDATE, heartbeats every ten seconds, marks its own rows aborted if it died mid-job, exits between jobs when the backend code changed, and is kept alive by a scheduled task with a restart supervisor.
The frontend is a Vite + React 19 single-page app with Recharts, hand-written CSS with design tokens, dark mode as a token swap, and a table twin for every chart.
data
Two databases addressed by URL: the read-only corpus (SQLite locally, Postgres copy when deployed) and the app store (users, saved jobs, operations queue, log lines, worker heartbeats, AI notes and enrichment). The migration script upserts and is safe to re-run; the worker syncs new and changed rows after every scrape and on demand.
Flyer images come from the scraper's local cache, an S3-compatible bucket the worker fills, or the Instagram CDN URL as a last resort, in that order.
deployment
Two Vercel projects from one repository. The backend is zero-config Python: Vercel detects the
FastAPI app from requirements.txt and loads the entry module, no rewrite rules. The frontend
is a static build with every path rewritten to index.html. Long-running work stays on the
worker because functions are capped at 300 seconds.
05
Key Findings
Engineering decisions
- The API never runs the scraper. Clicking "Run" writes a queued row; a worker on the machine that has the tools claims it with a conditional UPDATE, runs the subprocess, and streams state, log lines, and a heartbeat back into the same Postgres database.
- The corpus stays read-only everywhere. Locally the API opens the scraper's SQLite with mode=ro; the deployed API reads a Postgres copy that the worker refreshes after every scrape with incremental upserts.
- One filter definition is shared by the job list, statistics, and analytics endpoints, so every chart describes exactly the rows on screen.
- Post images are served through a cache proxy and an S3-compatible bucket with presigned URLs, because Instagram CDN URLs expire within days.
- AI features are a separate layer with a mock provider for tests and per-user daily caps. Without a key the features report themselves as unconfigured and the rest of the app is unaffected.
- Separate what runs where. The deployed API can never execute the scraper, so it only queues work. A worker on the machine with the tools claims jobs, runs them as subprocesses exactly as their Makefiles do, and writes progress into the shared database. This is what lets a Vercel deployment start and watch a scrape it could not run itself.
- The corpus is somebody else's. The scraper owns
job_postsandruns. The app opens the SQLite file read-only at the driver level and keeps a Postgres copy for the deployed API, refreshed incrementally with a 24-hour overlap. - One filter object drives the page. The job list, the statistics, and the analytics share one filter definition, so the numbers in every chart always describe the rows in the list.
- AI is optional and bounded. Everything the model produces is stored in the app database, never in the corpus. Quotas are per user per day because the provider's free tier is per organisation.
06
Challenges and Solutions
Challenges
- Vercel functions are capped at 300 seconds, so scrapes, enrichment, and syncs cannot live in the API.
- A deployed API has no access to the scraper's SQLite file or its multi-gigabyte image cache.
- Neon's transaction pooler breaks psycopg's automatic prepared statements.
- The scraper may be writing to its database while the API reads it.
Solutions
- Queue-and-worker split with conditional claims, a stop flag the worker polls, self-marking of aborted jobs on restart, and a scheduled-task supervisor that restarts the worker.
- A resumable migration script plus post-scrape sync to Postgres; the worker uploads images to the bucket and the API redirects to them.
- The engine disables prepared statements for any pooled connection URL.
- SQLite opened with mode=ro so the driver, not the code, refuses writes.
07
Result
A working dashboard with a queue-driven operations page, a Postgres-backed deployment, and AI features that degrade to "not configured" rather than failing. Tests run against a mock AI provider without any key. Usage numbers and uptime are not tracked.
Outcomes on record
- Ten API areas (auth, jobs, stats, analytics, meta, runs, saved, ops, admin, ai) behind one FastAPI app with Swagger documentation.
- Tests run against the mock AI provider without any API key.
- Deployed as two Vercel projects (Python API and static frontend) with Neon Postgres.
08
Stack
Languages
Infrastructure and IaC
Databases and storage
Messaging and streaming
Frontend
Tooling and platforms
AI and ML