Skip to main content

Storage

Careers Engine separates application logic from persistent data. Its operational state is maintained in a dedicated careers-data repository, independently of the main careers-engine repository. The storage layer currently uses JSON files to preserve state across workflow executions. Each file serves a distinct purpose: maintaining known opportunities, tracking publication history, or holding opportunities submitted through the manual queue.

Storage Layout

The persistent data repository contains the following files: Each destination has its own publication history, allowing the publishing pipeline to determine independently which opportunities still need to be delivered to Discord or Telegram. These files represent different parts of the system’s state and are not interchangeable.

How Storage Fits into the System

The ingestion pipeline populates the known-job collection, while the Queue Opportunity workflow provides a separate path for manually submitted opportunities. The publishing pipeline uses the relevant stored state to distribute opportunities and maintain publication history. For more detail, see the Ingestion Pipeline and Publishing Pipeline.

Job Records — jobs.json

jobs.json contains normalized job records produced by automated ingestion. Source adapters and parsers convert upstream listings into the canonical Job model before the records are persisted. This allows the application to process opportunities consistently regardless of whether they originated from Markdown tables or JSON datasets. The storage layer loads existing records, checks incoming jobs against known identifiers, and persists changes when new records are discovered. Each job receives a deterministic SHA-256 identifier derived from its company, role, location, and application URL. This identifier supports duplicate detection across ingestion runs, provided those fields remain unchanged. See the Ingestion Pipeline for the complete data flow and normalization process.

Destination-Specific Publication History

Careers Engine maintains separate publication histories for each destination:
  • delivery-history/discord.json tracks opportunities successfully published to Discord.
  • delivery-history/telegram.json tracks opportunities successfully published to Telegram.
A job may exist in jobs.json without having been published to either destination. The publishing pipeline compares job identifiers against each destination’s history to determine which opportunities still need to be delivered there. This separation allows an opportunity to be published to one destination without incorrectly treating it as published to every destination. For example, a job recorded in Discord history but absent from Telegram history can still be selected for Telegram publication. Successful Telegram deliveries are recorded individually as they occur, helping preserve delivery state when a later message encounters an error. Publication history still depends on successful persistence of the updated data, so it does not provide an absolute exactly-once delivery guarantee. See the Publishing Pipeline for the complete publication lifecycle.

Manual Opportunity Queue — queue.json

Not every opportunity necessarily originates from an automated source. Careers Engine also supports a manual submission path through the Queue Opportunity workflow. queue.json stores manually submitted opportunities for this workflow. It provides a place for manually supplied listings to enter the system without requiring an upstream source adapter to be implemented for each opportunity. This complements automated ingestion:
  • Automated ingestion retrieves listings from supported external sources.
  • Manual queuing allows opportunities to be submitted explicitly through the queue workflow.
  • Publishing distributes opportunities through the configured publishing mechanism.
The queue is a distinct input path, not a replacement for jobs.json or either destination’s publication history. The publishing workflow attempts delivery to Discord and Telegram independently. Queue state is cleared only when the publishing run completes without destination failures, so a failure on either destination can leave queued opportunities available for a subsequent run. The exact queue-processing and removal behavior is determined by the Queue Opportunity workflow implementation; the presence of queue.json alone does not imply that every queued item has already been published.

Repository Separation

Careers Engine uses separate repositories for application code and persistent operational data. The application repository defines how data is processed; the data repository preserves state between executions. GitHub Actions checks out the data repository when a workflow needs to read or update persistent state. When changes need to be saved, the workflow commits and pushes the updated files. This provides persistence across otherwise isolated workflow runs without requiring a continuously running database server.

Configuration

The storage directory is controlled by the DATA_DIR environment variable.
When DATA_DIR is not configured, the application defaults to a local data directory. Deployment workflows can point it to the checked-out careers-data repository. Keeping file paths derived from a single configuration value allows local development and automated execution to use different storage locations without changing application logic. For setup instructions, see Configuration and GitHub Actions.

Why JSON and Git-Based Persistence?

The current approach keeps the infrastructure relatively simple for the project’s workload.
  • Straightforward representation: Job records and workflow state can be stored as JSON.
  • Low operational overhead: No separate database service is required.
  • Version history: Git records changes to persistent data.
  • Workflow compatibility: GitHub Actions can load, update, and commit state during scheduled or manual executions.
  • Separation of concerns: Application code and generated operational state are managed independently.
These are trade-offs suited to the current implementation, not a claim that JSON is optimal for every scale.

Limitations and Future Considerations

The storage design has several constraints:
  • Concurrent workflows can contend when pushing changes to the same data repository.
  • JSON files are less suitable for complex queries and large datasets than a database.
  • Git-based persistence does not provide database transactions.
  • Exact-field identifiers may fail to recognize semantically identical listings with inconsistent metadata.
  • Each destination’s publication history must remain consistent with actual delivery outcomes.
  • A message may reach a destination before a later failure prevents the corresponding state from being persisted, so retries can potentially produce duplicates.
  • Queue processing must handle failures and state transitions carefully to avoid losing or repeatedly processing manually submitted opportunities.
If the system’s workload grows, possible improvements include stronger concurrency control, transactional persistence, more sophisticated deduplication, and a database backend such as PostgreSQL.

Next step

Storage provides the persistent state shared by automated ingestion, manual submissions, and publishing. Continue with Company Branding to understand how company identity and branding information are handled across job opportunities.

Company Branding