Admir Misini

Architecture Overview

CareerHolo is really two decoupled subsystems sharing one PostgreSQL database: a request-serving API and an independently scheduled ingestion pipeline. They never call each other directly; they only meet at the database:

  ┌────────────────────────────────────────────────────┐
  │              FastAPI  (api container)              │
  │    serves built React/Vite frontend + REST API     │
  │    Jobs · Search · Analytics · Skills · Trends     │
  └────────────────────────────────────────────────────┘
         reads/writes            cache & rate-limit
               │                          │
               ▼                          ▼
  ┌────────────────────────┐ ┌────────────────────────┐
  │       PostgreSQL       │ │         Redis          │
  └────────────────────────┘ └────────────────────────┘
               ▲
     writes scraped jobs
               │
  ┌────────────────────────────────────────────────────┐
  │               Scrapy Spider Network                │
  │       Greenhouse · Lever · Ashby · Workable        │
  │            Recruitee · SmartRecruiters             │
  │                RemoteOK · The Muse                 │
  └────────────────────────────────────────────────────┘
                             ▲
                        subprocess
                             │
  ┌────────────────────────────────────────────────────┐
  │                   Celery Worker                    │
  └────────────────────────────────────────────────────┘
                             ▲
                      scheduled by
                             │
  ┌────────────────────────────────────────────────────┐
  │       Celery Beat  (hourly / daily / weekly)       │
  └────────────────────────────────────────────────────┘

Core Components

• Scrapy Spiders – Parallelized web crawlers that fetch job listings from 8+ sources
• PostgreSQL Database – Persistent storage with optimized indexing for analytics
• Redis – Distributed queue, task broker, and rate limiter
• Celery + Beat – Asynchronous task scheduler for background jobs
• FastAPI – High-performance REST API with Pydantic validation
• React/Vite Frontend – Modern dashboard for searching and analyzing jobs

Key Technologies & Why They Were Chosen

Scrapy for Web Crawling

Scrapy is the backbone of CareerHolo's data ingestion:

• Parallelization: Uses ThreadPoolExecutor to run multiple spiders concurrently, dramatically increasing throughput
• Resilience: Built-in retry logic and auto-throttling prevent overwhelming target servers
• Flexibility: Custom pipelines for cleaning, tagging, filtering, and storing data

Example:

class GreenhouseSpider(BaseATSSpider):
    name = "greenhouse"
    env_var_name = "GREENHOUSE_COMPANIES"

    def start_requests(self):
        for company in self._load_companies():
            url = (
                f"https://boards-api.greenhouse.io/v1/boards/"
                f"{company}/jobs?content=true"
            )
            yield scrapy.Request(
                url, callback=self.parse_jobs, cb_kwargs={"company": company}
            )

    def parse_jobs(self, response, company):
        for job in json.loads(response.text).get("jobs", []):
            yield JobItem(
                title=job.get("title", "").strip(),
                company=job.get("company_name") or company,
                url=job.get("absolute_url"),
                description=job.get("content", "")
            )

PostgreSQL + Full-Text Search

Rather than relying on external search services, CareerHolo leverages PostgreSQL's native capabilities:

• TSVector & GIN Indexing: Ultra-fast full-text search across job titles, companies, and descriptions
• Optimized Analytics: Normalized job_skills table avoids expensive JSONB aggregations
• Composite Indexing: Strategic indexes on (is_active, created_at) speed up filtering and pagination

Example:

SELECT js.skill, COUNT(*) as demand_count
FROM job_skills js
JOIN jobs j ON js.job_id = j.id
WHERE j.search_vector @@ to_tsquery('python & fastapi')
GROUP BY js.skill
ORDER BY demand_count DESC;

Celery for Asynchronous Processing

Four scheduled cycles keep data fresh and system health high:

• Scraping Cycle (Hourly) – run_all_spiders fetches new jobs from all sources
• Liveness Cycle (Hourly) – run_liveness_check verifies job URLs are still active
• Discovery Cycle (Daily) – run_discovery scans company domains for new ATS links
• Slug Seeding Cycle (Weekly) – run_seed_company_slugs ingests curated company lists (YC, remoteintech.company)

Example:

@celery_app.task(name="workers.tasks.run_all_spiders")
def run_all_spiders_task():
    subprocess.run([sys.executable, "scripts/run_all_spiders.py"], check=True)

celery_app.conf.update(
    beat_schedule={
        "scrape-jobs-every-hour": {
            "task": "workers.tasks.run_all_spiders",
            "schedule": 3600.0,
        },
        # ...liveness, discovery, and weekly slug-seeding entries
    },
)

Feature Highlights

• Multi-Source Ingestion: Supports 8+ job sources including ATS platforms, aggregators, and custom discovery
• Advanced Analytics: Hiring trends, skill heatmaps, salary statistics, geo-distribution, and more
• Skill Extraction: Deterministic, high-speed extraction of required skills from job descriptions
• Full-Text Search: Lightning-fast searches across 30k+ jobs using PostgreSQL
• Rate Limiting & Resilience: Redis-backed rate limiting protects public endpoints

Example: FastAPI Endpoint with Caching

@router.get("/skills/top")
def top_skills(request: Request, limit: int = 20, db: Session = Depends(get_db)):
    def build():
        return query_top_skills(db, limit=limit)

    # Redis-backed cache keyed by route + query params, with
    # ETag / If-None-Match support for free 304s on repeat polls
    return cached_json_response(
        request, "analytics:skills:top", ttl_seconds=900, builder=build
    )

Performance & Scalability

CareerHolo's design prioritizes performance at every layer:

• Connection Pooling: FastAPI uses SQLAlchemy connection pools to reuse database connections
• Caching Strategy: Redis caches frequently-accessed analytics to eliminate redundant computation
• Batch Processing: The scraping pipeline buffers jobs (200 items or 15s, whichever first) and upserts them in batches to minimize database round trips
• Optimized Indexing: PostgreSQL GIN indexes enable sub-second full-text lookups
• Async-First Design: FastAPI's native async/await support handles thousands of concurrent requests

Real-World Use Cases

For Job Seekers: Understand which companies are hiring now and what skills are most valuable

For Recruiters: Find candidates matching specific skill sets across remote-first companies

For Market Researchers: Analyze salary trends, geographic hiring patterns, and emerging skill clusters

Lessons Learned

• Data Quality Over Quantity: Multi-stage pipelines ensure only relevant jobs are indexed
• Normalized Data Wins: Storing skills in a dedicated, indexed job_skills table instead of unpacking a JSON array per row cut analytics query times from 2+ seconds to <500ms
• Async-First Architecture: Decoupling scraping from the API prevented bottlenecks
• Dependency Pinning Matters: Clear version documentation prevents painful debugging cycles

What's Next?

CareerHolo is actively evolving with planned features including:

• Machine Learning NLP-based job classification and duplicate detection
• Skill-based recommendations for job seekers
• Real-time email alerts for saved searches
• Browser extension for one-click job saving
• Premium API access for recruiters and researchers

Conclusion

CareerHolo demonstrates how modern Python tooling (Scrapy, FastAPI, PostgreSQL, and Celery) can combine to build a sophisticated, production-ready platform. It's not just a job aggregator; it's a data pipeline, analytics engine, and API service rolled into one.

Whether you're analyzing hiring trends, building recruitment tools, or exploring the job market, CareerHolo shows that building scalable, performant systems is achievable with open-source technologies and thoughtful architecture.

Visit CareerHolo to get started.

Tech Stack Summary:

• Languages: Python 3.11, React/JavaScript, SQL
• Web: FastAPI, Pydantic, SQLAlchemy 2.0
• Scraping: Scrapy, Twisted
• Async: Celery, Redis
• Database: PostgreSQL 16, Alembic migrations
• Deployment: Docker, Docker Compose