OpenAI 计算机操作
计算机操作让模型使用浏览器或桌面界面,填写表单、测试用户流程或完成应用中的任务。
| 任务 | 起点 |
|---|---|
| 通过截图与鼠标键盘操作自己的浏览器或桌面 | 使用Responses的computer工具和 ComputerToolset |
| 读取自己浏览器的页面文本、使用控件和标签页 | 使用 BrowserToolset,由驱动将选定操作暴露为函数工具 |
| 使用OpenAI托管浏览器 | 参见Agents API计算机操作 |
只获取不需交互的公共信息时,使用网页搜索。
使用SDK
SDK在Python或Node中运行Responses工具循环。应用提供驱动,负责读取环境或执行允许操作;会话、权限和清理由应用负责。
连接托管环境
也可使用合作方的浏览器或桌面。有些直接提供驱动,有些提供需映射到SDK的控件。按提供方文档配置并核对包版本。
| 环境 | 集成 |
|---|---|
| Browserbase | Stagehand驱动与托管会话,参见集成指南及Python、TypeScript示例 |
| Browser Use | Python库驱动,或用云集成接入已有Playwright代码 |
| Daytona | ComputerToolset 加桌面沙箱,参见Python指导及Agents SDK示例 |
| E2B Desktop | 在桌面沙箱中运行操作或代码,参见指南、Python和TypeScript示例 |
设置会话超时;失败也要关闭或释放。清理失败时检查提供方控制台,并用其网络策略限制可访问站点与资源。
运行第一个任务
使用Python 3.11或以上,或Node.js 22.6或以上,并创建、导出API密钥。示例启动临时本地浏览器并调用Responses,可能产生费用。
在新目录安装依赖与Chromium:
来源示例1(sh)
## [中文] 建立虚拟环境并安装固定SDK和Playwright,再安装Chromium。
## [中文] Python环境安装命令。
## Python
python3 -m venv .venv
.venv/bin/python -m pip install openai==3.31.0 playwright==1.61.0
.venv/bin/python -m playwright install chromium
## [中文] Node环境安装命令。
## Node
npm install openai@7.33.0 playwright@1.61.1
npx playwright install chromium
Windows使用 py -3 替代 python3,使用 .venv\Scripts\python.exe 替代 .venv/bin/python。
截图与点击示例。 模型观察虚构取件页面截图,点击显示取件详情。驱动仅允许截图与对该控件左键点击,阻止页面网络请求,结束时关闭浏览器。脚本最多发起六次Responses请求,并检查页面后才报告成功;项目需有支持computer工具的模型访问权。保存为 pickup.py 或 pickup.mjs:
来源示例2(javascript)
程序中的自然语言译释39:
取件详情控件位置已改变,或该点击不被允许。
程序中的自然语言译释40:
本练习不允许该操作。
程序中的自然语言译释41:
取件详情仍处于关闭状态。
程序中的自然语言译释42:
先截图,打开Show pickup details(显示取件详情),报告柜号与取件码。只能使用截图和左键点击。
// [中文] 取件驱动只允许截图及目标控件内左键点击;执行点击前重新验证目标,最终检查页面并关闭浏览器。
import OpenAI, { ComputerToolset } from "openai";
import { chromium } from "playwright";
class PickupDriver {
constructor(page) {
this.page = page;
}
async screenshot() {
const image = await this.page.screenshot({ type: "png" });
return {
type: "computer_screenshot",
image_url: "data:image/png;base64," + image.toString("base64"),
};
}
async allowed(action) {
if (action.type === "screenshot") return true;
if (
action.type !== "click" ||
action.button !== "left" ||
action.keys?.length
)
return false;
const box = await this.page.locator("summary").boundingBox();
return Boolean(
box &&
box.x <= action.x &&
action.x < box.x + box.width &&
box.y <= action.y &&
action.y < box.y + box.height
);
}
async click(action) {
// [中文] 批准后页面可能发生变化,点击执行前再次核对并绑定原目标。
// 审批后页面可能已变化,因此要重新定位点击目标。
if (!(await this.allowed(action)))
throw new Error("Pickup details moved or the click is not allowed.");
await this.page.mouse.click(action.x, action.y);
}
refuse(action) {
throw new Error(`This exercise does not allow ${action.type}.`);
}
double_click = this.refuse;
drag = this.refuse;
keypress = this.refuse;
move = this.refuse;
scroll = this.refuse;
type = this.refuse;
wait = this.refuse;
}
async function main() {
const client = new OpenAI({ timeout: 30_000, maxRetries: 0 });
const browser = await chromium.launch({ env: {}, chromiumSandbox: true });
try {
const page = await browser.newPage({
viewport: { width: 1024, height: 768 },
});
await page.route("**/*", (route) => route.abort());
await page.setContent(
"<title>Parcel pickup</title><main>" +
"<h1>Parcel pickup</h1><p>Order #1042 is ready.</p>" +
"<details><summary>Show pickup details</summary>" +
"<p>Locker 18. Pickup code: 6249.</p></details></main>"
);
const driver = new PickupDriver(page);
const tools = new ComputerToolset(driver);
try {
const response = await tools.run(
client,
{
model: "gpt-6.1-sol",
store: false,
input:
"Take a screenshot. Open 'Show pickup details' and report the locker and pickup code. Use only screenshots and left clicks.",
},
{ beforeAction: (action) => driver.allowed(action), maxSteps: 6 }
);
if ((await page.locator("details").getAttribute("open")) === null) {
throw new Error("Pickup details are still closed.");
}
console.log(
"Verified pickup details:",
await page.locator("details p").innerText()
);
console.log("Answer:", response.output_text);
} finally {
tools.close();
}
} finally {
await browser.close();
}
}
await main();
来源示例3(python)
本段复用的中文释义:译释39、译释40、译释41、译释42。
## [中文] 取件驱动只允许截图及目标控件内左键点击;执行点击前重新验证目标,最终检查页面并关闭浏览器。
import asyncio
import base64
from openai import AsyncComputerToolset, AsyncOpenAI
from playwright.async_api import async_playwright
class PickupDriver:
def __init__(self, page):
self.page = page
async def screenshot(self):
image = await self.page.screenshot(type="png")
return {
"type": "computer_screenshot",
"image_url": "data:image/png;base64," + base64.b64encode(image).decode(),
}
async def allowed(self, action):
if action.type == "screenshot":
return True
if action.type != "click" or action.button != "left" or action.keys:
return False
box = await self.page.locator("summary").bounding_box()
return bool(
box
and box["x"] <= action.x < box["x"] + box["width"]
and box["y"] <= action.y < box["y"] + box["height"]
)
async def click(self, action):
# [中文] 批准后页面可能发生变化,点击执行前再次核对并绑定原目标。
# 审批后页面可能已变化,因此要重新定位点击目标。
if not await self.allowed(action):
raise RuntimeError("Pickup details moved or the click is not allowed.")
await self.page.mouse.click(action.x, action.y)
async def refuse(self, action):
raise RuntimeError(f"This exercise does not allow {action.type}.")
double_click = drag = keypress = move = scroll = type = wait = refuse
async def main():
async with (
async_playwright() as playwright,
AsyncOpenAI(timeout=30, max_retries=0) as client,
):
browser = await playwright.chromium.launch(env={}, chromium_sandbox=True)
try:
page = await browser.new_page(viewport={"width": 1024, "height": 768})
await page.route("**/*", lambda route: route.abort())
await page.set_content(
"<title>Parcel pickup</title><main>"
"<h1>Parcel pickup</h1><p>Order #1042 is ready.</p>"
"<details><summary>Show pickup details</summary>"
"<p>Locker 18. Pickup code: 6249.</p></details></main>"
)
driver = PickupDriver(page)
tools = AsyncComputerToolset(driver)
try:
response = await tools.run(
client,
{
"model": "gpt-6.1-sol",
"store": False,
"input": "Take a screenshot. Open 'Show pickup details' and report the locker and pickup code. Use only screenshots and left clicks.",
},
before_action=driver.allowed,
max_steps=6,
)
if await page.locator("details").get_attribute("open") is None:
raise RuntimeError("Pickup details are still closed.")
print(
"Verified pickup details:",
await page.locator("details p").inner_text(),
)
print("Answer:", response.output_text)
finally:
tools.close()
finally:
await browser.close()
if __name__ == "__main__":
asyncio.run(main())
运行 .venv/bin/python pickup.py 或 node pickup.mjs。成功打印 Verified pickup details: Locker 18. Pickup code: 6249.(已核验取件详情:18号柜,取件码6249),随后显示模型回答。待处理API安全检查会使此示例停止;只有应用能处理时才添加人工批准回调。
浏览器控件示例。 模型在访客证表单填写姓名Mira并点击Create badge(创建访客证)。驱动只暴露页面读取、输入和点击;确认回调仅允许该表单控件及姓名Mira。脚本检查页面结果后再报告成功。保存为 badge.py 或 badge.mjs:
来源示例4(javascript)
程序中的自然语言译释43:
为Mira创建访客证。
程序中的自然语言译释44:
访客证未成功创建,后面附实际结果。
// [中文] 访客证驱动通过state和read_page报告控件,confirm仅允许Mira及指定控件,结束时验收页面。
import OpenAI from "openai";
import { BrowserToolset } from "openai/helpers/browser";
import { ResponsesToolRunner } from "openai/helpers/responses-tool-runner";
import { chromium } from "playwright";
class BadgeForm {
constructor(page) {
this.page = page;
}
// [中文] 读取当前标签状态;这里报告观察结果,不授予操作权限。
async state() {
return {
tabs: [
{
id: "badge",
title: await this.page.title(),
url: this.page.url(),
active: true,
},
],
};
}
// [中文] 读取页面文字并返回结构化控件引用,供后续定位。
async read_page(_args) {
return {
version: 1,
text: await this.page.locator("body").innerText(),
elements: [
{ ref: "name", role: "textbox", label: "Name" },
{ ref: "save", role: "button", label: "Create badge" },
],
omitted: [],
};
}
async type(args) {
await this.page.locator("#name").fill(args.text);
}
async click(_args) {
await this.page.locator("#save").click();
}
// [中文] 按当前页面及固定目标参数判断是否允许;只有明确批准才继续。
async confirm({ member, arguments: args }) {
if (
this.page.url() !== "about:blank" ||
(args.tab_id !== null && args.tab_id !== "badge")
) {
return false;
}
return (
(member === "type" && args.target === "name" && args.text === "Mira") ||
(member === "click" && args.target === "save")
);
}
}
async function main() {
const client = new OpenAI({ timeout: 30_000, maxRetries: 0 });
const browser = await chromium.launch({ env: {}, chromiumSandbox: true });
try {
const page = await browser.newPage();
await page.route("**/*", (route) => route.abort());
await page.setContent(`
<title>Visitor badge</title>
<label>Name <input id="name"></label>
<button id="save" onclick="document.querySelector('output').innerText =
'Badge ready: ' + document.querySelector('#name').value">Create badge</button>
<output></output>
`);
const driver = new BadgeForm(page);
const tools = new BrowserToolset(driver, {
members: ["read_page", "type", "click"],
confirm: driver.confirm.bind(driver),
});
const response = await new ResponsesToolRunner({ browser: tools }).run(
client,
{
model: "gpt-6-astra",
store: false,
input: "Create a visitor badge for Mira.",
},
{ maxSteps: 6 }
);
const result = await page.locator("output").innerText();
if (result !== "Badge ready: Mira") {
throw new Error(`The badge was not created: ${JSON.stringify(result)}`);
}
console.log(result);
console.log("Answer:", response.output_text);
} finally {
await browser.close();
}
}
await main();
来源示例5(python)
本段复用的中文释义:译释43、译释44。
## [中文] 访客证驱动通过state和read_page报告控件,confirm仅允许Mira及指定控件,结束时验收页面。
import asyncio
from openai import AsyncOpenAI
from openai.lib.browser import AsyncBrowserToolset
from openai.lib.responses_tool_runner import AsyncResponsesToolRunner
from playwright.async_api import async_playwright
class BadgeForm:
def __init__(self, page):
self.page = page
# [中文] 读取当前标签状态;这里报告观察结果,不授予操作权限。
async def state(self):
return {
"tabs": [
{
"id": "badge",
"title": await self.page.title(),
"url": self.page.url,
"active": True,
}
]
}
# [中文] 读取页面文字并返回结构化控件引用,供后续定位。
async def read_page(self, args):
return {
"version": 1,
"text": await self.page.locator("body").inner_text(),
"elements": [
{"ref": "name", "role": "textbox", "label": "Name"},
{"ref": "save", "role": "button", "label": "Create badge"},
],
"omitted": [],
}
async def type(self, args):
await self.page.locator("#name").fill(args["text"])
async def click(self, args):
await self.page.locator("#save").click()
# [中文] 按当前页面及固定目标参数判断是否允许;只有明确批准才继续。
async def confirm(self, context):
args = context["arguments"]
if self.page.url != "about:blank" or args["tab_id"] not in (None, "badge"):
return False
return (
context["member"] == "type"
and args["target"] == "name"
and args["text"] == "Mira"
) or (context["member"] == "click" and args["target"] == "save")
async def main():
async with (
async_playwright() as playwright,
AsyncOpenAI(timeout=30, max_retries=0) as client,
):
browser = await playwright.chromium.launch(env={}, chromium_sandbox=True)
try:
page = await browser.new_page()
await page.route("**/*", lambda route: route.abort())
await page.set_content("""
<title>Visitor badge</title>
<label>Name <input id="name"></label>
<button id="save" onclick="document.querySelector('output').innerText =
'Badge ready: ' + document.querySelector('#name').value">Create badge</button>
<output></output>
""")
driver = BadgeForm(page)
tools = AsyncBrowserToolset(
driver, members=["read_page", "type", "click"], confirm=driver.confirm
)
response = await AsyncResponsesToolRunner(browser=tools).run(
client,
{
"model": "gpt-6-astra",
"store": False,
"input": "Create a visitor badge for Mira.",
},
max_steps=6,
)
result = await page.locator("output").inner_text()
if result != "Badge ready: Mira":
raise RuntimeError(f"The badge was not created: {result!r}")
print(result)
print("Answer:", response.output_text)
finally:
await browser.close()
if __name__ == "__main__":
asyncio.run(main())
运行相应脚本,成功输出 Badge ready: Mira(访客证已就绪:Mira)及模型回答。API拒绝工具或Chromium无法启动时参见运行排错,点击偏移时参见截图与坐标。
检查模型权限、成本与数据
- 模型:浏览器函数工具要求支持Responses和函数调用,返回图片时还需图像输入;截图式操作须核对模型Tools栏的computer能力及项目访问权。
- 成本:每次请求可产生输入输出词元费用,截图计为图像输入;核对价格、图像成本及请求用量。托管环境可能另收费。
- 数据:SDK会将返回的页面文字与截图发给OpenAI。
store: false不取消所有数据保留;核对端点数据控制、图像输入例外及环境提供方政策。
查看并批准变更
可尝试Python预约演示或Node演示:填写虚构预约,提交前询问批准,检查保存字段和确认结果。用户可观看、接管、交回或停止。两者均提供真实浏览器中的无API本地演练。观察桌面SDK操作可用可视桌面演示。
调整驱动
将示例页面换成自己控制的环境。浏览器用 members 只暴露任务所需操作;桌面实现必需方法并拒绝不支持操作。运行结束前保持会话,之后关闭应用创建的资源。
页面内容会影响模型,应使用新浏览器配置或隔离桌面,限制网络,避免凭证出现在文字与截图中。变更前审批,报告成功前检查实际保存结果。驱动参考说明方法协议、审批、URL策略和恢复。
对有界选择切换到Decisions
模型需选择动作并生成参数时使用Responses。应用已知所有允许动作及完整参数时才用Decisions,例如预先定义 route_billing、route_technical、review。Decisions只选择,应用仍需批准、执行并验收;页面变化后应丢弃旧选项。参见Decisions。
在预约演示中,用有Decisions权限的密钥,设置 OPENAI_DECISIONS_MODEL 并重启服务器,开始新运行再选择Decisions。
直接使用API
需要自行管理工具循环时直接用Responses。应用提供环境、执行模型请求。可选:模型通过PyAutoGUI或Playwright生成操作代码,单次调用组合动作、循环与条件;或使用computer工具,接收结构化鼠标键盘操作。GPT-6 Astra推荐代码执行方式,computer仍可用。
已有函数调用或MCP界面工具也可保留,差异见自有UI工具。
使用代码执行
给模型一个接受脚本的函数工具,应用在隔离浏览器或桌面执行,返回文本及截图。调用之间保持环境,使模型能延续工作。
运行示例应用
CUA示例应用有JavaScript/Playwright与Python/PyAutoGUI实现、内置任务和共用控制台。按选定实现的说明在隔离环境安装;选择场景运行;检查动作、截图和最终状态。安装、桌面权限与支持环境以README为准,接入真实账户前先阅读安全要求。
连接自己的运行时
下例Python、Ruby、Go向PyAutoGUI桌面运行时发送Python代码,JavaScript用Playwright操作浏览器。均使用普通函数工具,以原始 call_id 返回文本或图像。
execute_in_sandbox 或 executeInSandbox 由应用实现,负责保持会话、限制执行及权限;它们是集成片段,不是直接运行示例应用的代码。
来源示例6(python)
程序中的自然语言译释45:
在持久桌面中运行Python,变量跨调用保留;PyAutoGUI操作为同步。
可用对象包括pyautogui、time、log(value)、display(PIL_image)。
操作前用display(pyautogui.screenshot())检查屏幕。使用截图坐标,执行少量动作后再检查。
截图保留在内存,并保持PyAutoGUI的fail-safe开启。
程序中的自然语言译释46:
任务达到20个响应的上限,请检查最后结果。
## [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
import json
import uuid
from openai import OpenAI
from openai.types.responses import (
FunctionToolParam,
ResponseInputParam,
)
def run_computer_use(endpoint, prompt, model="gpt-6-astra"):
client = OpenAI()
session_id = str(uuid.uuid4())
tools: list[FunctionToolParam] = [
{
"type": "function",
"name": "exec_py",
"description": (
"Run Python in a persistent desktop. Variables persist across calls. "
"PyAutoGUI operations are synchronous. Available: pyautogui, time, "
"log(value), and display(PIL_image). Inspect the screen with "
"display(pyautogui.screenshot()) before acting. Use screenshot "
"coordinates and check the screen after a short group of actions. "
"Keep screenshots in memory and PyAutoGUI's fail-safe enabled."
),
"parameters": {
"type": "object",
"properties": {"code": {"type": "string"}},
"required": ["code"],
"additionalProperties": False,
},
"strict": True,
}
]
next_input: ResponseInputParam = [{"role": "user", "content": prompt}]
previous_response_id = None
for turn in range(20):
response = client.responses.create(
model=model,
tools=tools,
input=next_input,
previous_response_id=previous_response_id,
)
if response.status != "completed":
raise RuntimeError(f"Response stopped with status: {response.status}")
calls = [item for item in response.output if item.type == "function_call"]
if not calls and any(
item.type == "message" and item.phase != "commentary"
for item in response.output
):
print(response.output_text)
return
if turn == 19:
raise RuntimeError(
"The task reached the 20-response limit. Inspect the last result."
)
next_input = []
for call in calls:
if call.name != "exec_py":
raise ValueError(f"Unexpected tool: {call.name}")
code = json.loads(call.arguments)["code"]
output = execute_in_sandbox(code, session_id, endpoint)
next_input.append(
{
"type": "function_call_output",
"call_id": call.call_id,
"output": output,
}
)
previous_response_id = response.id
来源示例7(javascript)
程序中的自然语言译释107:
在持久浏览器中运行JavaScript。可用Playwright的browser、context、page对象,以及console.log(value)和display(base64Image)。
把可复用变量保存在globalThis。操作前查看截图,少量动作后再检查屏幕,截图保留在内存。
异步操作使用顶层await;display()返回图像,console.log()返回简洁文字。上下文视口为1440×900。
本段复用的中文释义:译释46。
// [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
import { randomUUID } from "node:crypto";
import OpenAI from "openai";
async function runComputerUse(endpoint, prompt, model = "gpt-6-astra") {
const client = new OpenAI();
const sessionId = randomUUID();
const tools = [
{
type: "function",
name: "exec_js",
description: `Run JavaScript in a persistent browser. Available: Playwright's
browser, context, and page objects; console.log(value); and display(base64Image).
Save reusable variables on globalThis. Inspect a screenshot before acting and
check the screen after a short group of actions. Keep screenshots in memory.
Use top-level await for async operations. Return images with display() and concise
text with console.log(). The context viewport is 1440x900.`,
parameters: {
type: "object",
properties: { code: { type: "string" } },
required: ["code"],
additionalProperties: false,
},
strict: true,
},
];
let nextInput = [{ role: "user", content: prompt }];
let previousResponseId;
for (let turn = 0; turn < 20; turn++) {
const response = await client.responses.create({
model,
tools,
input: nextInput,
previous_response_id: previousResponseId,
reasoning: { effort: "low" },
});
// [中文] 响应未完整完成时停止,避免执行部分生成的请求。
if (response.status !== "completed") {
throw new Error(`Response stopped with status: ${response.status}`);
}
const calls = response.output.filter(
(item) => item.type === "function_call"
);
if (
calls.length === 0 &&
response.output.some(
(item) => item.type === "message" && item.phase !== "commentary"
)
) {
console.log(response.output_text);
return;
}
if (turn === 19) {
throw new Error(
"The task reached the 20-response limit. Inspect the last result."
);
}
nextInput = [];
for (const call of calls) {
if (call.name !== "exec_js")
throw new Error(`Unexpected tool: ${call.name}`);
const { code } = JSON.parse(call.arguments);
const output = await executeInSandbox(code, sessionId, endpoint);
nextInput.push({
type: "function_call_output",
call_id: call.call_id,
output,
});
}
previousResponseId = response.id;
}
}
来源示例8(ruby)
本段复用的中文释义:译释45。
## [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
require "json"
require "openai"
require "securerandom"
def run_computer_use(endpoint, prompt)
client = OpenAI::Client.new
session_id = SecureRandom.uuid
tools = [
{
type: :function,
name: "exec_py",
description: "Run Python in a persistent desktop. Variables persist across calls. PyAutoGUI operations are synchronous. Available: pyautogui, time, log(value), and display(PIL_image). Inspect the screen with display(pyautogui.screenshot()) before acting. Use screenshot coordinates and check the screen after a short group of actions. Keep screenshots in memory and PyAutoGUI's fail-safe enabled.",
parameters: {
type: :object,
properties: { code: { type: :string } },
required: ["code"],
additionalProperties: false
},
strict: true
}
]
next_input = []
next_input << {
role: :user,
content: prompt
}
history = {}
20.times do |turn|
response = client.responses.create(
model: "gpt-6-astra", tools: tools, input: next_input, previous_response_id: history[:id]
)
raise "Response stopped with status: #{response.status}" unless response.status == OpenAI::Responses::ResponseStatus::COMPLETED
calls = response.output.grep(OpenAI::Responses::ResponseFunctionToolCall)
if calls.empty? && response.output.any? { |item| item.is_a?(OpenAI::Responses::ResponseOutputMessage) && item.phase != :commentary }
puts(response.output_text)
return response
end
raise "The task reached the 20-response limit" if turn == 19
next_input.clear
calls.each do |call|
raise "Unexpected tool: #{call.name}" unless call.name == "exec_py"
code = JSON.parse(call.arguments).fetch("code")
raise "Expected Python source text" unless code.is_a?(String)
output = execute_in_sandbox(code, session_id, endpoint)
next_input << {
type: :function_call_output,
call_id: call.call_id,
output: output
}
end
history[:id] = response.id
end
end
来源示例9(go)
本段复用的中文释义:译释45。
// [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
func runComputerUse(ctx context.Context, client openai.Client, endpoint, prompt string, terminal *bufio.Scanner, output io.Writer) error {
session := make([]byte, 16)
if _, err := rand.Read(session); err != nil {
return err
}
sessionID := hex.EncodeToString(session)
tool := responses.ToolParamOfFunction("exec_py", map[string]any{
"type": "object", "properties": map[string]any{"code": map[string]any{"type": "string"}},
"required": []string{"code"}, "additionalProperties": false,
}, true)
tool.OfFunction.Description = openai.String("Run Python in a persistent desktop. Variables persist across calls. " +
"PyAutoGUI operations are synchronous. Available: pyautogui, time, " +
"log(value), and display(PIL_image). Inspect the screen with " +
"display(pyautogui.screenshot()) before acting. Use screenshot " +
"coordinates and check the screen after a short group of actions. " +
"Keep screenshots in memory and PyAutoGUI's fail-safe enabled.")
params := responses.ResponseNewParams{
Model: "gpt-6-astra", Tools: []responses.ToolUnionParam{tool},
Input: responses.ResponseNewParamsInputUnion{OfString: openai.String(prompt)},
}
for turn := 0; turn < 20; turn++ {
response, err := client.Responses.New(ctx, params)
if err != nil {
return err
}
if response.Status != "completed" {
return fmt.Errorf("response stopped with status: %s", response.Status)
}
var calls []responses.ResponseFunctionToolCall
finalMessage := false
for _, item := range response.Output {
if item.Type == "function_call" {
calls = append(calls, item.AsFunctionCall())
}
if item.Type == "message" && item.AsMessage().Phase != "commentary" {
finalMessage = true
}
}
if len(calls) == 0 && finalMessage {
fmt.Fprintln(output, response.OutputText())
return nil
}
if turn == 19 {
return fmt.Errorf("task reached the 20-response limit; inspect the last result")
}
nextInput := responses.ResponseInputParam{}
for _, call := range calls {
if call.Name != "exec_py" {
return fmt.Errorf("unexpected tool: %s", call.Name)
}
var arguments struct {
Code *string `json:"code"`
}
if err := json.Unmarshal([]byte(call.Arguments), &arguments); err != nil {
return err
}
if arguments.Code == nil {
return fmt.Errorf("exec_py requires code")
}
observations, err := executeInSandbox(ctx, endpoint, sessionID, *arguments.Code, terminal, output)
if err != nil {
return err
}
nextInput = append(nextInput, responses.ResponseInputItemUnionParam{
OfFunctionCallOutput: &responses.ResponseInputItemFunctionCallOutputParam{
CallID: openai.String(call.CallID),
Output: responses.ResponseInputItemFunctionCallOutputOutputUnionParam{OfResponseFunctionCallOutputItemArray: observations},
},
})
}
params.Input = responses.ResponseNewParamsInputUnion{OfInputItemList: nextInput}
params.PreviousResponseID = openai.String(response.ID)
}
return nil
}
完整适配器及输出形态见连接执行服务。该服务属于你的应用,不是OpenAI托管端点。
保持状态并返回观察结果
保持浏览器或桌面会话;持久Python或JavaScript命名空间也可保留变量。工具定义要描述可用对象与辅助函数。界面未知时先返回当前截图,执行少量动作后再截图核对。图像保存在内存,采用 detail: original。缩图后应把模型坐标映射回环境坐标,参见截图。
API会话与执行环境是两套状态。会话保留调用和结果,应用保留对应环境;续接响应不会恢复浏览器、登录或运行时变量。
使用computer工具
需要结构化动作而非生成代码时采用。按同一示例应用设置选择Native模式,并选支持computer的模型。流程是发送任务、执行动作、返回截图。示例页面有Show filters(显示筛选)和搜索框,可按自己界面调整。
环境与动作处理器见集成方案。
发送任务
在 tools 启用 computer,说明期望结果:
来源示例10(javascript)
程序中的自然语言译释47:
检查Filters(筛选)面板是否打开;未打开则点击Show filters。随后在搜索框输入penguin。使用computer工具进行界面交互。
程序中的自然语言译释69:
检查Filters面板是否打开。
// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
import OpenAI from "openai";
const client = new OpenAI();
const response = await client.responses.create({
model: "gpt-6.1-sol",
tools: [{ type: "computer" }],
input:
"Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",
});
console.log(JSON.stringify(response.output, null, 2));
来源示例11(python)
本段复用的中文释义:译释47、译释69。
## [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
from openai import OpenAI
client = OpenAI()
response = client.responses.create(
model="gpt-6.1-sol",
tools=[{"type": "computer"}],
input="Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",
)
print(response.output)
来源示例12(go)
本段复用的中文释义:译释47、译释69。
// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
package main
import (
"context"
"fmt"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/responses"
)
func main() {
client := openai.NewClient()
response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{
Model: "gpt-6.1-sol",
Tools: []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},
Input: responses.ResponseNewParamsInputUnion{OfString: openai.String("Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.")},
})
if err != nil {
panic(err)
}
fmt.Println(response.Output)
}
来源示例13(java)
程序中的自然语言译释129:
必要时打开Filters面板,再搜索penguin;使用computer工具进行界面交互。
// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.responses.ResponseCreateParams;
import java.util.List;
import java.util.Map;
ResponseCreateParams params =
ResponseCreateParams.builder()
.model("gpt-6.1-sol")
.input(
"Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.")
.putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))
.build();
client.responses().create(params).output().forEach(System.out::println);
来源示例14(ruby)
本段复用的中文释义:译释129。
## [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
require "openai"
client = OpenAI::Client.new
response = client.responses.create(
model: "gpt-6.1-sol",
input: "Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.",
tools: [{ type: :computer }]
)
puts(response.output)
执行请求动作
computer_call 含有序 actions。下例选择搜索框并输入 penguin:
来源示例15(json):以下保留协议载荷、配置、返回示意或语法定义;省略号及片段不代表完整可运行程序。
{
"output": [
{
"type": "computer_call",
"call_id": "call_002",
"actions": [
{ "type": "click", "button": "left", "x": 405, "y": 157 },
{ "type": "type", "text": "penguin" }
],
"status": "completed"
}
]
}
处理器按序把允许的动作映射到浏览器或操作系统输入,再截取更新画面。动作包括 click、double_click、drag、move、scroll、keypress、type、wait、screenshot。
首次可能只有截图请求,此时不改变UI,只返回当前画面。调用 status: completed 仅表示模型生成该调用完毕,应用仍需执行。
键位、拖拽及修饰键映射见动作处理器。
返回截图
返回 call_id 对齐的 computer_call_output,使用 previous_response_id 续接模型会话。
来源示例16(javascript)
// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
import OpenAI from "openai";
const client = new OpenAI();
async function sendComputerScreenshot(response, callId, screenshotBase64) {
const output = {
type: "computer_screenshot",
image_url: `data:image/png;base64,${screenshotBase64}`,
detail: "original",
};
return await client.responses.create({
model: "gpt-6.1-sol",
tools: [{ type: "computer" }],
previous_response_id: response.id,
input: [
{
type: "computer_call_output",
call_id: callId,
output,
},
],
});
}
来源示例17(python)
## [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
from openai import OpenAI
client = OpenAI()
def send_computer_screenshot(response, call_id, screenshot_base64):
return client.responses.create(
model="gpt-6.1-sol",
tools=[{"type": "computer"}],
previous_response_id=response.id,
input=[
{
"type": "computer_call_output",
"call_id": call_id,
"output": {
"type": "computer_screenshot",
"image_url": f"data:image/png;base64,{screenshot_base64}",
"detail": "original",
},
}
],
)
来源示例18(go)
// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
package main
import (
"context"
"fmt"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/responses"
)
func main() {
client := openai.NewClient()
response, err := sendComputerScreenshot(client, "resp_abc123", "call_abc123", "<base64 bytes here>")
if err != nil {
panic(err)
}
fmt.Println(response.Output)
}
func sendComputerScreenshot(client openai.Client, responseID string, callID string, screenshotBase64 string) (*responses.Response, error) {
screenshot := responses.ResponseComputerToolCallOutputScreenshotParam{
ImageURL: openai.String("data:image/png;base64," + screenshotBase64),
}
screenshot.SetExtraFields(map[string]any{"detail": "original"})
return client.Responses.New(context.Background(), responses.ResponseNewParams{
Model: "gpt-6.1-sol",
Tools: []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},
PreviousResponseID: openai.String(responseID),
Input: responses.ResponseNewParamsInputUnion{OfInputItemList: responses.ResponseInputParam{
responses.ResponseInputItemParamOfComputerCallOutput(callID, screenshot),
}},
})
}
来源示例19(java)
// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.responses.ResponseComputerToolCallOutputScreenshot;
import com.openai.models.responses.ResponseCreateParams;
import com.openai.models.responses.ResponseInputItem;
import java.util.List;
import java.util.Map;
String responseId = "resp_abc123";
String computerCallId = "call_abc123";
String screenshotBase64 = "<base64 bytes here>";
ResponseCreateParams params =
ResponseCreateParams.builder()
.model("gpt-6.1-sol")
.input(
ResponseCreateParams.Input.ofResponse(
List.of(
ResponseInputItem.ofComputerCallOutput(
ResponseInputItem.ComputerCallOutput.builder()
.callId(computerCallId)
.output(
ResponseComputerToolCallOutputScreenshot.builder()
.imageUrl("data:image/png;base64," + screenshotBase64)
.putAdditionalProperty("detail", JsonValue.from("original"))
.build())
.build()))))
.previousResponseId(responseId)
.putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))
.build();
client.responses().create(params).output().forEach(System.out::println);
来源示例20(ruby)
## [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
require "openai"
client = OpenAI::Client.new
response = client.responses.create(
model: "gpt-6.1-sol",
previous_response_id: "resp_abc123",
input: [
{
type: :computer_call_output,
call_id: "call_abc123",
output: {
type: :computer_screenshot,
image_url: "data:image/png;base64,<base64 bytes here>",
detail: :original
}
}
],
tools: [{ type: :computer }]
)
puts(response.output)
保持环境可用,遵循相同截图与状态规则。
重复到不再返回computer调用,检查其余输出是回答、求助还是其他工具,再在应用验收;本例应打开筛选面板,搜索框含 penguin。循环骨架见重复操作循环。
安全运行
- 隔离浏览器或虚拟机,限制站点与操作到任务所需范围。
- 页面、文档和工具结果均为不可信内容,不能赋予权限或覆盖用户指令。
- 购买、传输数据、破坏性或难以撤销操作应由用户掌控;向表单输入敏感信息也属于传输。
- 设置步数、时间或成本上限,支持取消,检查实际结果,而非只信最终回答。
具体审批、人工接管和指令例子参见确认与同意。