24761 Chap.10 AI-Assisted Analysis and Critical Evaluation
AI-Assisted Analysis and Critical Evaluation
A tool, not an analyst
The second report asks you to analyse one more secondary source with a generative AI tool, then show your own judgement about the output. You stay in the role of insights analyst for the brand from the first report, and the new evidence has to support, strengthen, refine or challenge the issue you named there.
The rubric gives as many points to evaluating the output as to producing the analysis, so the checking is the core of the task.
Prompt, check, refine
The brief favours an iterative process over a single prompt.
This chapter works a sequence on invented forum posts about a clinic chain: a vague first request, a count the tool overstated, a theme it missed, and the checks that caught each problem before it reached the report.
Writing it up
The chapter maps the five report sections and their word guides, gives five tests for any AI output, and shows how the evaluation leads into the primary research you recommend next.
It also explains why user-generated data needs care: the people who post may not be a random sample of customers.
What this chapter covers
- 01
Generative AI in data-driven insights
- 02
Iterative prompting and the evidence log
- 03
Checking AI counts, quotes and themes
- 04
Bias and limits in forum and comment data
- 05
Support, refine or challenge the earlier issue
- 06
Section word guides for the second report
Worked example · free
Verify an AI count of cost posts about a vet clinic chain
- 1Overstatement: 140 minus 88 is 52 posts, so the tool counted 52 more than exist, about 59% above the true number of mentions (52 divided by 88).
- 1Share mentioning cost: 88 of 320 is 27.5% of all posts.
- 1Share complaining about cost: 88 minus 31 praise leaves 57, and 57 of 320 is 17.8%.
- 1Evaluation sentence: the tool inflated cost mentions and did not separate praise from complaint, so the report uses the hand-checked 17.8% and lists the correction in the appendix.
Key terms
- Generative AI
- Software such as Copilot that produces new text, summaries or charts in answer to a prompt, rather than only retrieving stored information.
- Iterative Prompting
- Refining a series of prompts, each narrower or differently framed than the last, and checking every output before the next step.
- Professional Judgement
- The analyst’s own reasoned decision about whether an output is accurate, relevant and useful enough to rely on.
- AI Output Evaluation
- Testing what a tool produced for quality, relevance, accuracy, completeness and usefulness before it informs a decision.
AI-Assisted Analysis and Critical Evaluation FAQ
How do I check an AI analysis of forum posts or open-ended comments?
Treat the tool’s first answer as a lead. Recount any number it gives against the raw rows, look up every quote it offers, and read a random sample of the text yourself to find themes it missed. Record each correction beside the request that produced it, because that record shows your professional judgement in the appendix.
What does critically evaluating AI output mean in practice?
It means testing the output rather than reporting it. Check whether counts and quotes match the source, whether the themes bear on your issue, what the tool left out, and whether the data itself is biased, for example because people who post online may over-represent strong experiences. Then say how those checks changed your conclusion.
Do my AI prompts count towards the report word limit?
No. The appendix sits outside the word count, and the brief expects it to hold the relevant prompts, the AI outputs and supporting evidence from the extra source you analysed. Showing several refined prompts and how you judged each answer is viewed more favourably than presenting one response.
Which AI tool should I use for the second report?
The brief gives Microsoft Copilot as an example, and the assignment pages recommend it because the information you enter remains confidential. Whatever you use, declare it in APA format, upload a true transcript of the chat with your submission and keep drafts and search history for four months.
Can AI-generated sentiment scores be trusted?
Only after checking. The Week 5 lecture explains that sentiment is scored either with a fixed word list or by a model trained on large text collections, and both can misread sarcasm or a comment that praises one thing and criticises another. Read a sample of scored comments and report the error you find.
Assessment move
Before the report is due, run a small rehearsal: take thirty public forum posts on a topic unrelated to your brand, ask an AI tool for themes and counts, and check every number by hand. Note where it overstated, what it missed and how a sharper prompt fixed it. Keep that rehearsal log as a model for your real appendix.
Then draft the evaluation section first, before the description, so the word guide of about 200 words goes to judgement rather than summary, and end it by naming the limits your primary research should close. Throughout, keep a one-line note of every correction you make, because those notes become the strongest evidence of judgement in the appendix.
Working through AI-Assisted Analysis and Critical Evaluation in 24761? Sia is AskSia’s AI Marketing tutor — ask any 24761 AI-Assisted Analysis and Critical Evaluation question and get a clear, step-by-step explanation grounded in how 24761 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.