How the AI-powered data pipeline behind the Skill Dashboard works

Every day we fetch new US job postings from a popular jobs API and keep each one whose title contains both “AI” and “Engineer”.

Searching for “AI Engineer” alone would miss most of them: real titles look like “AI Solutions Engineer” or “Staff Engineer - Agentic AI”. A posting that shows up more than once is counted once.

A title isn’t enough, so an LLM (gpt-5.6-luna) reads each posting and decides whether it’s really an AI engineering role. We follow Chip Huyen’s book AI Engineering: an AI Engineer’s main job is building applications on top of LLMs and other foundation models.

Many postings with “AI” in the title are really for ML Engineers, who train and fine-tune models, or MLOps, who run the infrastructure. The LLM judges the responsibilities, not the title, and leaves those postings out.

From every posting classified as AI Engineering, an LLM (gpt-5.6-luna) extracts each skill it names and whether that skill is must-have, nice-to-have or only-mentioned.

Names are matched against the skills already known, so “AWS” and “Amazon Web Services” count as one skill. Each skill also gets a category, such as programming languages or cloud.