Indonesia Job Data Scraping System
A local Python pipeline that collects publicly visible Instagram job-vacancy posts across Indonesia's 38 provinces, reads the flyer images with OCR, extracts and normalises the vacancy fields, deduplicates within and across runs, and stores structured output as JSON, JSON Lines, CSV, and SQLite.
- Role
- Sole developer. Pipeline design, extraction, OCR, cleaning, tests.
- Context
- Personal project. Runs locally, no login and no access-control bypass. Its output feeds the Job Listing Dashboard (case 002).
- Period
- 2026-08 → ongoing (Reconnaissance verified live on 2026-08-11; still maintained.)
- Evidence
- ArchitectureDocumentationTechnical breakdown
01
The Case
Job vacancies in Indonesia are posted as Instagram flyers by hundreds of regional "loker" accounts. The information a job seeker needs sits in an image caption and, more often, inside the image itself. There is no API, no search for anonymous clients, and every account has its own layout.
The task was to turn that stream into structured, de-duplicated records per province without logging in, without bypassing any access control, and without inventing a single field.
02
Evidence
README covering the pipeline, schema, compliance, and endpoint-status notes
document
Reconnaissance report with sanitised probe evidence
document
Extraction metrics report per run (tools/extraction_metrics.py)
log
03
Architecture
Text version of the diagram
Sources and configuration
-
Instagram public surfaces
external
Signed-out profile, feed, and post pages. Search and hashtag endpoints are login-walled and never used.
-
Configuration
settings.yaml, 38 provinces and 478 cities, keyword lexicon, category taxonomy, 82 seed accounts.
Pipeline stages
-
Discovery
Python
Seed accounts per province plus username templates, each verified anonymously.
-
Fetch
curl_cffi
Timeline paging, newest first, stops at the start date. Chrome TLS impersonation.
-
Parse
pydantic
Instagram JSON to typed RawPost objects; unexpected shapes raise.
-
Classify
Keyword-and-weight scoring producing is_job_post and a confidence.
-
OCR
RapidOCR
Flyer images read locally with the English PP-OCRv4 recogniser; raw text kept verbatim.
-
Normalise
Caption and OCR merged into title, company, salary, contacts; province and city from five weighted signals.
-
Clean (LLM + regex)
Groq API
LLM reader validated against the source text, regex fallback, phone gate.
-
Deduplicate
SQLite
post_id, canonical URL, content hash; within and across runs.
Outputs
-
Store
Raw JSON, processed JSON, JSON Lines, CSV, SQLite, entry-ready TXT per province.
-
Job Listing Dashboard
external
Case 002 reads the SQLite corpus and syncs it to Postgres.
Connections
- Configuration → Discovery
- Instagram public surfaces → Fetch · HTTPS (HTTP)
- Discovery → Fetch
- Fetch → Parse
- Parse → Classify
- Classify → OCR
- OCR → Normalise
- Normalise → Clean (LLM + regex)
- Clean (LLM + regex) → Deduplicate
- Deduplicate → Store
- Store → Job Listing Dashboard · read-only SQLite
04
Technical Breakdown
application
Stages: discovery, fetch, parse, date filter, classify, OCR, normalise, deduplicate, store, plus
a separate cleaning stage. The OCR engine is RapidOCR (ONNX, pure pip) with the English
PP-OCRv4 recogniser; Tesseract is a drop-in alternative. OCR never fails a run: a missing engine
or a corrupt image downgrades one post to skipped or failed with its URLs intact.
The LLM reader runs on Groq's free tier, tries four models in order, cools a model down on a 429, caches answers by source text, and falls back to regex when no model can answer.
data
Thirty-seven fields per post, defined once in a pydantic model. Contact information is structured (e-mails, phones, raw phones, WhatsApp links). Locations are resolved from five weighted signals with a confidence score; the Instagram geotag alone is never trusted because it reflects the poster, not the job site.
Outputs per run: untouched raw API payloads (audit trail), pretty-printed processed JSON per province, an append-only JSON Lines export, CSV, SQLite with a cross-run dedup index, and entry-ready text files for the downstream form-filling tool.
security
The scraper reads only what Instagram serves to a signed-out browser. It never logs in, solves
a CAPTCHA, or works around an access-control response; a 403, CAPTCHA, or login redirect is
terminal by policy. The robots.txt notice is logged at startup and an enforce mode makes the
run exit with zero requests. Whether running it complies with Instagram's Terms of Use is the
operator's decision, stated in the README.
05
Key Findings
Engineering decisions
- HTTP-only collection with browser TLS impersonation (curl_cffi, chrome124 profile). No login, no CAPTCHA or access-control bypass, and a robots.txt notice with warn, enforce, and ignore modes.
- Nine-stage pipeline (discovery, fetch, parse, date filter, classify, OCR, normalise, deduplicate, store) with one module per stage and a per-stage entry point for offline debugging.
- 37-field pydantic output schema with extra="forbid". Unextractable fields stay null, which a dedicated "nothing is fabricated" test enforces.
- Three-key deduplication (post id, canonical URL, content hash) applied within a run and across runs through a SQLite index.
- Cleaning stage reads each post with a small LLM on Groq's free tier, validates every number and e-mail digit-for-digit against the source text, and falls back to regex extractors when no model can answer.
- Reconnaissance before code. A dedicated recon phase probed every public surface and
recorded what Instagram refused. The design follows those results: account-based discovery,
the public REST feed over GraphQL (whose
doc_idrotates every few weeks), and TLS-fingerprint impersonation because Instagram gates on the fingerprint before it reads headers. - One module per stage. A renamed field or a new pagination cursor touches one file. Every stage also has its own entry point, so a problem can be reproduced offline against captured data instead of re-hitting the network.
- Schema as a contract. The output model forbids unknown fields and a test asserts that no extractor ever fabricates a value. Raw OCR text is stored verbatim next to the parsed view.
- A reader, not more regex. Four thousand lines of patterns could not tell a heading from a field. The cleaning stage now asks a small model for structured JSON and then refuses any number, e-mail, title, or employer that does not survive validation against the source text.
06
Challenges and Solutions
Challenges
- Instagram's search and hashtag endpoints are login-walled, so keyword search is impossible for an anonymous client.
- Flyer text lives in images, and the default OCR model dropped the spaces between Latin-script words.
- Regex extraction mistook headings and slogans ("We Are Hiring") for job titles and employers.
- Endpoint behaviour changed twice (login walls on 2026-08-12 and 2026-09-04), shrinking anonymous reach.
Solutions
- Account-based discovery. Curated seed accounts per province plus username templates, each verified anonymously; 82 accounts across 37 of 38 provinces.
- RapidOCR with the English PP-OCRv4 recogniser plus vocabulary-based re-spacing. Raw OCR text is always preserved next to the parsed fields.
- An LLM reader with strict validation against the source text and a phone gate that routes rows without a mobile number to a review file.
- A fail-loud error taxonomy. A 404 is per-input, a 403 or login redirect is terminal by policy, an unexpected JSON shape stops the run. Fallback to the signed-out profile surface when the feed endpoint closed.
07
Result
The pipeline runs daily across provinces, with 314 offline tests guarding the extraction path. Cross-run deduplication was verified live by re-running an identical scrape and writing zero records. On a 24-post hand-labelled set the LLM cleaning path produced entry-ready rows for every post; the regex-only path managed about one in five.
Reach is bounded by Instagram: since the 2026-09-04 login wall, an anonymous client sees the twelve newest posts per account.
Outcomes on record
- 314 offline tests using trimmed real API responses as fixtures.
- Re-running the same scrape produced zero duplicate records (cross-run deduplication verified live).
- On a 24-post hand-labelled gold set the LLM cleaning path produced entry-ready rows for every post, while the regex-only path managed about one in five.
Planned, not built
- Revive snowball discovery if Instagram exposes a related-accounts surface again. Planned
08
Stack
Data and analytics
Databases and storage
Tooling and platforms
AI and ML