Aurum
Download

LLM Application Engineering

LLM features that hold up in production: a chosen model and spend caps, versioned prompts and schemas, a golden eval set, retrieval and tool calling done carefully, and the reviews that catch drift, cost creep and bad outputs.

Free · Mac (Apple silicon) and Windows · v1.0.0 · Other platforms

Then open the Marketplace in the app, search for LLM Application Engineering, and click Install. It goes into your own files, and works offline from then on.

What it installs.

1

Pillar

1

Area

50

Projects

196

Starter tasks

⚙️ Software Engineering

LLM Application Engineering

Building features on large language models, including prompts, retrieval, tool calling, evaluations, cost control and handling unreliable outputs.

  1. Problem fit check before building with an LLM
  2. Model provider and model shortlist
  3. API keys, quotas and spend caps
  4. Golden dataset of fifty real examples
  5. Prompts kept as versioned files in the repo
  6. Structured output contract with schema validation
  7. Call wrapper with timeouts, retries and fallback
  8. Model call logging with personal data redaction
  9. Provider data retention and training terms review
  10. Weekly grading of sampled production outputs
  11. Eval suite run in CI on prompt changes
  12. Cost per task tracking for model usage
  13. Model deprecation and version change watch
  14. Thumbs-down feedback triage for AI features
  15. Latency budget for model-backed screens
  16. Regression set grown from production failures
  17. Abuse and unsafe output report review
  18. Quarterly model re-benchmark on the golden set
  19. Prompt writing fundamentals practice
  20. Tokens and context window mechanics
  21. Embeddings and vector search fundamentals
  22. Tool calling and function schema design
  23. LLM-as-judge grader design and calibration
  24. Chunking and retrieval parameter experiments
  25. Prompt injection attack and defence practice
  26. Sampling settings and nondeterminism study
  27. Retrieval pipeline for document question answering
  28. Prompt caching for repeated context
  29. Routing between a small and a large model
  30. Streaming responses in the interface
  31. Hosted API versus self-hosted open-weight model
  32. Orchestration framework or plain SDK decision
  33. Grounding checks and citations to reduce hallucination
  34. Batch processing for offline LLM workloads
  35. Vector database selection
  36. Launch readiness review for an AI feature
  37. Planned migration to a new model version
  38. Red-team session before public launch
  39. Provider outage drill
  40. Investor or stakeholder demo of an AI prototype
  41. Six-week founder MVP built on an LLM
  42. Unit economics of an AI-powered product
  43. Inheriting an LLM feature someone else built
  44. Shared LLM practices for a product team
  45. AI feature with human review in a regulated domain
  46. Agent loop with tool use and hard limits
  47. Fine-tuning decision and first small fine-tune
  48. Multimodal pipeline for images and scanned documents
  49. Online experiments on prompt and model variants
  50. Permission-aware retrieval across users and tenants

Where it starts.

Each project arrives with its purpose, milestones and first tasks written out. Here is how the first three read.

Problem fit check before building with an LLM

Plenty of LLM features get built for problems that a lookup table, a regular expression or a search box would solve faster, cheaper and more reliably. Writing down the task, what a wrong answer costs and how a person would judge a good one tells you whether a model belongs in the design before anyone writes a prompt.

First tasks

  • Write the feature as one sentence naming its input and its output
  • List what happens to the user when the answer is wrong
  • Sketch the simplest non-LLM version and guess its accuracy
  • Record the go or no-go decision and share it with the team

Model provider and model shortlist

Picking whichever model was in the last blog post locks in price, latency and data terms you never compared. Scoring two or three providers on the same twenty real examples, with price per million tokens, context length, rate limits and data retention side by side, makes the choice defensible and easy to revisit.

First tasks

  • Pick twenty real inputs that cover easy, typical and awkward cases
  • Run each candidate model on all twenty with the same prompt
  • Fill in quality, latency and cost per task for each model
  • Name a primary and a fallback model and note why

API keys, quotas and spend caps

One leaked key or a runaway retry loop can spend a month's budget overnight, and most providers will happily let it. Separate keys per environment, a hard monthly cap, budget alerts at sensible thresholds and keys kept in the secrets manager rather than the repo turn that risk into an email.

First tasks

  • Run a secret scan over the repo for provider API keys
  • Issue separate keys for development, staging and production
  • Set a hard monthly cap and alerts at 50 and 80 percent
  • Compare month-to-date spend with the cap and the forecast @recurring(monthly:5)

Make it yours.

Install LLM Application Engineering, then tell Aurum the date you're aiming for. It checks whether the plan fits the weeks you actually have.

Free · Mac (Apple silicon) and Windows · v1.0.0 · Other platforms

Blueprints are general information to help you organise your own work. They are not medical, financial, legal or other professional advice. See the Terms.