System Architecture
At a high level, the system follows this flow:
Major Components
Upstream Sources
Sources provide the raw career opportunity data consumed by the engine. The current sources are:- SpeedyApply — provides internship and new-grad opportunities through Markdown files.
- Simplify — provides internship listings through a JSON dataset.
BaseSource interface so that the ingestion pipeline does not need to know the details of an individual upstream.
Source Adapters
A source adapter is responsible for communicating with an upstream source and converting its raw data into records that the engine can process. Source-specific concerns such as URLs, upstream files, and source-specific fields remain inside the corresponding source implementation. This keeps the core pipeline independent of individual upstream repositories.Parsers
Different sources expose data in different formats. The engine therefore separates fetching source data from parsing that data. The current parser implementations include:MarkdownTableParserJsonParser
MarkdownTableParser, while Simplify’s JSON dataset is processed by JsonParser.
The parser produces structured records that can then be converted into the canonical Job model.
Job Model
TheJob model provides the common representation used by the rest of the system.
Regardless of which upstream produced an opportunity, the pipeline works with the same model.
A job contains fields such as:
- company
- role
- location
- application URL
- employment type
- eligibility
- stipend
- deadline
- description
- priority
- discovery timestamp
Eligibility and Filtering
After source data has been converted into jobs, the pipeline applies eligibility filters. The current eligibility filter keeps opportunities whose location explicitly indicates India. The filtering layer is separate from source implementations so that eligibility rules can evolve without modifying individual source adapters.Storage
The engine stores its persistent dataset separately from the application source code. The current deployment uses thecareers-data repository as the data store.
The primary datasets include:
jobs.json— collected career opportunities.history.json— publishing history used to determine which jobs have already been published.
Publishing
Publishing is handled separately from ingestion. The publishing pipeline reads the stored jobs, determines which opportunities have not yet been published, formats them for the configured publisher, and sends them to the distribution channel. The current publisher is Discord. Publishing history is persisted so that subsequent executions do not repeatedly publish the same opportunities.Automation
The system is executed through GitHub Actions. The current automation consists of two independent workflows:-
Ingestion workflow
- collects opportunities from configured sources
- applies the ingestion pipeline
- updates
careers-data/jobs.json
-
Publishing workflow
- reads the stored jobs
- checks publishing history
- publishes new opportunities to Discord
- updates
careers-data/history.json
Design Principles
The architecture follows several principles.Modular Sources
Adding a new upstream should require implementing a source adapter and, when necessary, a parser rather than modifying the entire ingestion pipeline.Separation of Concerns
Fetching, parsing, modeling, filtering, persistence, and publishing are separate responsibilities.Idempotent Automation
Repeated ingestion and publishing runs should not continuously create duplicate records or repost already-published opportunities.Data Separation
Application code and persistent career data are maintained separately.Testable Components
Core components are implemented independently so that source adapters, parsers, filters, and other behavior can be tested without requiring the complete production workflow.Architecture Boundaries
The engine is divided into two primary operational paths.Ingestion
The ingestion path is responsible for discovering and storing opportunities.Publishing
The publishing path is responsible for distributing opportunities that have not previously been published.Repository Boundaries
The application code and persistent data are maintained in separate repositories.careers-engine
Contains the application itself, including:
- source adapters
- parsers
- filters
- models
- storage logic
- publishers
- tests
- GitHub Actions workflows
- documentation
careers-data
Contains the persistent datasets produced and consumed by the engine.
The primary files are:
jobs.jsonhistory.json
Extending the Architecture
The architecture is designed around interfaces and independent components. A new source can be introduced by implementing the source abstraction and connecting the appropriate parser to it. A source that exposes a format already supported by the engine can reuse an existing parser. A source using a different format can introduce a new parser without changing the rest of the ingestion pipeline. Similarly, additional filters or publishers can be introduced without requiring the existing source implementations to know about them. This keeps the core pipeline stable as the number of sources and distribution channels grows.Where to Go Next
The remaining architecture documentation describes each part of the system in greater detail:- Ingestion Pipeline — how opportunities move from upstream sources into persistent storage.
- Publishing Pipeline — how stored opportunities are selected and published.
- Storage — how persistent job and publishing state is maintained.
- Company Branding — how company branding and logos are handled.
- Employment Inference — how employment types are inferred from opportunity data.
- Repository Layout — how the source repository is organized.