IS5542 Chap.8 Measuring Value: KPIs, Evaluation and ROI
Measuring Value: KPIs, Evaluation and ROI
How a GenAI system is sensed
After the problem domains, the course asks how a team knows whether its application works, and answers with key performance indicators: quantifiable measures that give an objective view of how the model performs, how the system behaves and what the business gains.
The group project then asks for these measures in its results section.
Three tiers of KPI
Model performance covers accuracy and error rate, latency, a safety score for toxicity, bias and sensitive disclosure, groundedness in the supplied context, and how well instructions are followed.
System quality covers uptime, throughput, system latency including retrieval, the relevance and reuse of data, and environmental impact.
Business impact covers adoption, frequency of use, session length, satisfaction including staff concerns about job security and upskilling, cost savings and productivity gains.
Human and automated evaluation
Experts, end users and crowds can judge coherence, creativity and usefulness that metrics miss.
Automated metrics are repeatable at volume: overlap scores for text against references or human summaries, perplexity for how well a model predicts text, and a score for the quality and diversity of generated images.
Presenting the evidence
The project template allots time to results, evaluation and business impact, asking for KPIs, qualitative insights, a comparison with the current baseline and an estimate of return on investment.
A common trap
Teams often report the tier that looks best.
A fast, reliable system with low adoption, or an accurate model that nobody trusts, fails the business test; the course's three tiers exist so that a presentation shows where value is actually created and where it is lost.
What this chapter covers
- 01
Why KPIs answer the question of sensing
- 02
Model performance KPIs
- 03
System quality KPIs
- 04
Business impact KPIs
- 05
Human evaluation methods
- 06
Automated metrics for text and images
- 07
Baselines, ROI and the project presentation
Worked example · free
Choose the KPIs for a first-line HR assistant
- 2Choose a model performance KPI and say how it is measured.
- 2Choose a system quality KPI.
- 2Choose a business impact KPI.
- 3Explain how the three combine into a business case.
Key terms
- Key Performance Indicator
- A quantifiable measure tracking the progress and success of an initiative, used here at model, system and business level.
- Safety Score
- A model KPI assessing harmlessness, considering toxicity, bias and disclosure of sensitive information.
- Adoption Rate
- The share of intended users who actively use a GenAI application.
- Perplexity
- An automated measure of how well a language model predicts a sample of text; lower is better.
- Return on Investment
- The benefit of a solution, in money or other business terms, set against what it costs, estimated in the project presentation.
- Baseline
- The current performance of the process before the GenAI solution, against which results are compared.
Measuring Value: KPIs, Evaluation and ROI FAQ
Why is accuracy alone not enough to justify a GenAI rollout?
Accuracy is only the model tier. A system can be accurate and still fail because it is slow, unavailable or expensive, and a well-running system can still fail the business if nobody adopts it. The course measures all three tiers so that a rollout decision rests on the whole chain from model to money.
What should the results section of the group presentation include?
Quantitative results expressed as KPIs, qualitative insights such as observed changes in workflow, a comparison with the current situation or existing solutions, and a statement of business value with an estimate of return on investment. The template also suggests quantifying improvements wherever possible, for example as percentage gains or cost savings.
When should human evaluation be used instead of automated metrics?
When quality is subjective. Coherence, creativity and fluency are hard to score automatically, so experts, users or crowds judge them. Automated overlap and perplexity scores are cheaper and repeatable across large volumes, which makes them better for tracking changes between versions once human judgement has set the standard.
Exam move
For three applications from other chapters, a voice agent, a claims assistant and a marketing content tool, write two KPIs per tier and one sentence on how each would be measured. Then rehearse stating a baseline and a return-on-investment estimate aloud, since the group presentation gives that section four minutes.
Working through Measuring Value: KPIs, Evaluation and ROI in IS5542? Sia is AskSia’s AI Information Technology tutor — ask any IS5542 Measuring Value: KPIs, Evaluation and ROI question and get a clear, step-by-step explanation grounded in how IS5542 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
