对于LLM大模型应用来说，提示工程prompting engineering 是基本的，也是很重要的工具，对于模型智力和推理能力有着至关重要的影响。
从手工开发和维护prompt，一些框架开始创造性和自动的生成和优化prompt，例如React和Auto GPT等。斯坦福的项目 DSPy就是一个prompt优化的框架。
DSPy利用数据集迭代优化prompt，类似机器学习的过程，工作流程一般是：
1-收集训练数据
2-定义一个 program
3-选择评估方式和优化器
4-Compile！
import dspy
from dspy.datasets.gsm8k import GSM8K, gsm8k_metric
from dspy.teleprompt import BootstrapFewShot
from dspy.evaluate import evaluate
# Set up the LM
turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct', max_tokens=250,
api_key='your_OPENAI_API_KEY')
dspy.settings.configure(lm=turbo)
# 载入训练数据
gsm8k = GSM8K()
gsm8k_trainset, gsm8k_devset = gsm8k.train[:10], gsm8k.dev[:10]
# 定义 program
class CoT(dspy.Module):
def __init__(self):
super().__init__()
self.prog = dspy.ChainOfThought("question -> answer")
def forward(self, question):
return self.prog(question=question)
# 定义优化器
config = dict(max_bootstrapped_demos=4, max_labeled_demos=4)
teleprompter = BootstrapFewShot(metric=gsm8k_metric, **config)
# Compile！
optimized_cot = teleprompter.compile(CoT(), trainset=gsm8k_trainset)
# 效果评估
evaluate = evaluate(devset=gsm8k_devset, metric=gsm8k_metric, num_threads=4, display_progress=True, display_table=0)
evaluate(optimized_cot)
turbo.inspect_history(n=1)
生成的prompt为：
Given the fields `question`, produce the fields `answer`.
---
Follow the following format.
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Question: Sarah buys 20 pencils on Monday. Then she buys 18 more pencils on Tuesday. On Wednesday she buys triple the number of pencils she did on Tuesday. How many pencils does she have?
Reasoning: Let's think step by step in order to determine the total number of pencils. On Monday, Sarah bought 20 pencils. On Tuesday, she bought another 18, bringing the total to 38 pencils. Then on Wednesday, she bought triple the number of pencils she did on Tuesday, which would be 3 x 18 = 54 pencils. So the total number of pencils is 38 + 54 = 92.
Answer: 92
---
Question: Rookie police officers have to buy duty shoes at the full price of $85, but officers who have served at least a year get a 20% discount. Officers who have served at least three years get an additional 25% off the discounted price. How much does an officer who has served at least three years have to pay for shoes?
Reasoning: Let's think step by step in order to solve this problem. We can start by calculating the discount that an officer who has served at least three years can get. This would be 25% of the discounted price. Then, we can find the original discounted price, which is 85 - (85 * 20%) = 85 - 17 = 68. Finally, we can calculate the final price by applying the additional 25% discount on the discounted price of 68, resulting in a discount of 17 and a final price of 51.
Answer: 51
---
Question: The average score on last week's Spanish test was 90. Marco scored 10% less than the average test score and Margaret received 5 more points than Marco. What score did Margaret receive on her test?
Reasoning: Let's think step by step in order to produce the answer. First, we know that Marco scored 10% less than the average, which means he scored 90 - (90 * 0.1) = 90 * 0.9 = 81. Then, we know that Margaret scored 5 more points than Marco, which means she scored 81 + 5 = 86.
Answer: 86
---
Question: A third of the contestants at a singing competition are female, and the rest are male. If there are 18 contestants in total, how many of them are male?
Reasoning: Let's think step by step in order to find the number of male contestants. We know the total number of contestants is 18 and that a third of them are female. This means that there are 2/3 of the contestants are male. In order to find the number of male contestants, we can multiply 18 by 2 and divide by 3, giving us a total of 12 male contestants.
Answer: 12
---
Question: Trey is raising money for a new bike that costs $112. He plans to spend the next two weeks selling bracelets for $1 each. On average, how many bracelets does he need to sell each day?
Reasoning: Let's think step by step in order to find the average number of bracelets Trey needs to sell each day. We know that Trey needs to raise $112 and each bracelet costs $1. This means he needs to sell 112 bracelets in total. He has two weeks to sell them, which means he needs to sell 112/14 = 8 bracelets per day on average.
Answer: 8