A prompt that produced a clean depreciation rollforward in March can produce something noticeably different in September, with no change on your end. Nothing in the prompt broke. Three things around it moved:
The model changed. Providers update models, and the same wording can be interpreted differently. A prompt that said "summarize the variances" may now return more detail, less detail, or a different ordering.
The inputs changed. A new chart of accounts, a revised tariff, a new FERC order, a different export layout from your billing system. The prompt still references the old structure.
Your requirements changed. Materiality thresholds get revised, the board packet format gets updated, an auditor asks for a different level of support. The prompt still reflects last year's version of the task.
The fix is to treat prompts the way you treat active accounting schedules. Each one gets:
A version and a date. When it was last changed and last confirmed to work, the same way a lead schedule shows its preparer and review date.
A known-answer test. Keep one input where you already know the correct output, such as a prior-period reconciliation that was tied out and signed off. Rerun it whenever the model, the source data layout, or the requirements change. If the output no longer matches, the prompt needs review before it touches current-period work.
A change log. Record what was changed and why. When a result looks off in three months, you can see whether the prompt, the model, or the data moved.
An owner. One person is responsible for each prompt. A shared prompt that nobody maintains is the same problem as a shared spreadsheet that nobody maintains.
A reasonable cadence is a known-answer rerun at each quarter-end close, and again any time you see the model version change.
This takes less time than it sounds. For most utilities, the working library is ten to fifteen prompts. The goal is that when a prompt starts to drift, you find it with a test case and not in the middle of a rate filing.
Russ Hissom, CPA is a principal of UtilityEducation.com, an online training platform offering certified continuing education courses in accounting, rates, construction accounting, financial analysis, management and artificial intelligence applications for utilities.
Learn more at UtilityEducation.com or contact Russ at [email protected].