Comparative prompts
The information and exercises here draw on material developed for Module 2 of Arizona State University’s online undergraduate course on prompt engineering using ChatGPT.
Comparative prompts are designed to ask ChatGPT (or similar systems) to compare or rank options rather than generating a standalone response. Comparative prompts can help improve the quality and reliability of model outputs by leveraging the model’s (apparently emergent) ability to reason about the relative merits of different choices. You will explore best practices for crafting comparative prompts as well as the benefits and potential challenges associated with this approach.
The exercises below are designed to help develop your understanding and use of comparative prompts. They work best if using the paid ChatGPT Plus service from Open AI and GPT-4.
Overview
These exercises are designed to help you understand and use comparative prompt engineering. You will be given a task and asked to explore how you can formulate prompts that lead to ChatGPT providing and ranking multiple options. The exercises will encourage you to experiment with question formats, context, and instructions.
Exercise: Comparing Options and Analyses
1. Open a ChatGPT session. Make sure you are in GPT-4 mode.
2. Choose a topic. Select a topic that is relevant to your field of study or an area of interest. Make sure it has at least two distinct options or aspects that can be compared (e.g. if your topic was K-12 education, two distinct aspects might be home schooling versus class-based education).
3. Formulate a general prompt: Create a general prompt related to your chosen topic that covers the different options or aspects you’re interested in, but that doesn’t provide ChatGPT specific guidelines on how to answer. For instance, an example of a general prompt might be “Which renewable energy source is better: solar or wind?”, “Who is the better writer, Jane Austin or George Elliot?”, or “Which is more effective, home schooling or class-based schooling?” (These are just examples – please use your own prompt).
Note: These are prompts that are open ended and leave it up to ChatGPT to decide how best to answer. As a result, the responses are likely to be very general.
4. Transform your initial prompt into a comparative prompt: Modify your prompt from above to focus on comparing and ranking options or aspects based on specific criteria. For instance “Compare and rank solar and wind energy sources based on their efficiency, environmental impact, and cost-effectiveness.” Or “Compare Jane Austen and George Elliot based on their style and social relevance.” Or “Compare the pros and cons of home schooling versus class based schooling for students with learning difficulties in rural areas.” (These are just examples – use your own prompt)
5. Evaluate the responses: Compare the usefulness and specificity of generated responses, considering the following points: a) Which prompt generated a more informative and useful response? b) How did the comparative prompt help guide ChatGPT’s reasoning and focus on the specific criteria? c) Were there any limitations or challenges associated with the comparative prompt?
6. Iterate and refine: Based on your evaluation, refine your comparative prompt if necessary and test it again with ChatGPT. Repeat this process until you are satisfied with the response quality.
7. Submit documentation of your session: Submit a link to your session with ChatGPT below.
At the end of this exercise you should have a clearer sense of how to use comparative prompts.
Exercise: Ranking Options and Explainability
1. Open a ChatGPT session. Make sure you are in GPT-4 mode.
2. Choose a topic. Select a topic that is relevant to your field of study or an area of interest. Make sure it has at least three or four distinct options or aspects that can be compared. This can be the same as the topic in the previous assignment.
3. Formulate a ranking prompt. Create a prompt that asks ChatGPT to rank aspects of your topic against a specific criteria. For instance “List in order of potential habitability the planets of the solar system” or “Rank the sustainability of various forms of transportation from most to least” (these are just examples).
4. Explainability. Ask ChatGPT to explain the reasoning behind the ranking. If you disagree with it, ask for further clarification.
IMPORTANT: Please do not miss out this step in asking ChatGPT to explain the ranking. This is a critical part of using ChatGPT in smart ways.
At the end of this exercise you should be able to use ChatGPT to rank options and responses, and to probe the “reasoning” behind rankings. Interrogating ChatGPT on its reasoning is a powerful approach to understanding the userfulness and validity of responses — the reasoning may sometimes be suspect or plain wrong, but this ability to explore what led to responses allows you to both apply your own discernment and develop prompt engineering skills that lead to more relevant and trustworthy responses.