codecamp

OpenAI 计算机操作

计算机操作让模型使用浏览器或桌面界面,填写表单、测试用户流程或完成应用中的任务。

任务 起点
通过截图与鼠标键盘操作自己的浏览器或桌面 使用Responses的computer工具和 ComputerToolset
读取自己浏览器的页面文本、使用控件和标签页 使用 BrowserToolset,由驱动将选定操作暴露为函数工具
使用OpenAI托管浏览器 参见Agents API计算机操作

只获取不需交互的公共信息时,使用网页搜索。

使用SDK

SDK在Python或Node中运行Responses工具循环。应用提供驱动,负责读取环境或执行允许操作;会话、权限和清理由应用负责。

连接托管环境

也可使用合作方的浏览器或桌面。有些直接提供驱动,有些提供需映射到SDK的控件。按提供方文档配置并核对包版本。

环境 集成
Browserbase Stagehand驱动与托管会话,参见集成指南及Python、TypeScript示例
Browser Use Python库驱动,或用云集成接入已有Playwright代码
Daytona ComputerToolset 加桌面沙箱,参见Python指导及Agents SDK示例
E2B Desktop 在桌面沙箱中运行操作或代码,参见指南、Python和TypeScript示例

设置会话超时;失败也要关闭或释放。清理失败时检查提供方控制台,并用其网络策略限制可访问站点与资源。

运行第一个任务

使用Python 3.11或以上,或Node.js 22.6或以上,并创建、导出API密钥。示例启动临时本地浏览器并调用Responses,可能产生费用。

在新目录安装依赖与Chromium:

来源示例1(sh)

## [中文] 建立虚拟环境并安装固定SDK和Playwright,再安装Chromium。
## [中文] Python环境安装命令。
## Python
python3 -m venv .venv
.venv/bin/python -m pip install openai==3.31.0 playwright==1.61.0
.venv/bin/python -m playwright install chromium

## [中文] Node环境安装命令。
## Node
npm install openai@7.33.0 playwright@1.61.1
npx playwright install chromium

Windows使用 py -3 替代 python3,使用 .venv\Scripts\python.exe 替代 .venv/bin/python。

截图与点击示例。 模型观察虚构取件页面截图,点击显示取件详情。驱动仅允许截图与对该控件左键点击,阻止页面网络请求,结束时关闭浏览器。脚本最多发起六次Responses请求,并检查页面后才报告成功;项目需有支持computer工具的模型访问权。保存为 pickup.py 或 pickup.mjs:

来源示例2(javascript)

程序中的自然语言译释39:

取件详情控件位置已改变,或该点击不被允许。

程序中的自然语言译释40:

本练习不允许该操作。

程序中的自然语言译释41:

取件详情仍处于关闭状态。

程序中的自然语言译释42:

先截图,打开Show pickup details(显示取件详情),报告柜号与取件码。只能使用截图和左键点击。

// [中文] 取件驱动只允许截图及目标控件内左键点击;执行点击前重新验证目标,最终检查页面并关闭浏览器。
import OpenAI, { ComputerToolset } from "openai";
import { chromium } from "playwright";

class PickupDriver {
  constructor(page) {
    this.page = page;
  }

  async screenshot() {
    const image = await this.page.screenshot({ type: "png" });
    return {
      type: "computer_screenshot",
      image_url: "data:image/png;base64," + image.toString("base64"),
    };
  }

  async allowed(action) {
    if (action.type === "screenshot") return true;
    if (
      action.type !== "click" ||
      action.button !== "left" ||
      action.keys?.length
    )
      return false;
    const box = await this.page.locator("summary").boundingBox();
    return Boolean(
      box &&
      box.x <= action.x &&
      action.x < box.x + box.width &&
      box.y <= action.y &&
      action.y < box.y + box.height
    );
  }

  async click(action) {
    // [中文] 批准后页面可能发生变化,点击执行前再次核对并绑定原目标。
    // 审批后页面可能已变化,因此要重新定位点击目标。
    if (!(await this.allowed(action)))
      throw new Error("Pickup details moved or the click is not allowed.");
    await this.page.mouse.click(action.x, action.y);
  }

  refuse(action) {
    throw new Error(`This exercise does not allow ${action.type}.`);
  }

  double_click = this.refuse;
  drag = this.refuse;
  keypress = this.refuse;
  move = this.refuse;
  scroll = this.refuse;
  type = this.refuse;
  wait = this.refuse;
}

async function main() {
  const client = new OpenAI({ timeout: 30_000, maxRetries: 0 });
  const browser = await chromium.launch({ env: {}, chromiumSandbox: true });
  try {
    const page = await browser.newPage({
      viewport: { width: 1024, height: 768 },
    });
    await page.route("**/*", (route) => route.abort());
    await page.setContent(
      "<title>Parcel pickup</title><main>" +
        "<h1>Parcel pickup</h1><p>Order #1042 is ready.</p>" +
        "<details><summary>Show pickup details</summary>" +
        "<p>Locker 18. Pickup code: 6249.</p></details></main>"
    );
    const driver = new PickupDriver(page);
    const tools = new ComputerToolset(driver);
    try {
      const response = await tools.run(
        client,
        {
          model: "gpt-6.1-sol",
          store: false,
          input:
            "Take a screenshot. Open 'Show pickup details' and report the locker and pickup code. Use only screenshots and left clicks.",
        },
        { beforeAction: (action) => driver.allowed(action), maxSteps: 6 }
      );
      if ((await page.locator("details").getAttribute("open")) === null) {
        throw new Error("Pickup details are still closed.");
      }
      console.log(
        "Verified pickup details:",
        await page.locator("details p").innerText()
      );
      console.log("Answer:", response.output_text);
    } finally {
      tools.close();
    }
  } finally {
    await browser.close();
  }
}

await main();

来源示例3(python)

本段复用的中文释义:译释39、译释40、译释41、译释42。

## [中文] 取件驱动只允许截图及目标控件内左键点击;执行点击前重新验证目标,最终检查页面并关闭浏览器。
import asyncio
import base64

from openai import AsyncComputerToolset, AsyncOpenAI
from playwright.async_api import async_playwright


class PickupDriver:
    def __init__(self, page):
        self.page = page

    async def screenshot(self):
        image = await self.page.screenshot(type="png")
        return {
            "type": "computer_screenshot",
            "image_url": "data:image/png;base64," + base64.b64encode(image).decode(),
        }

    async def allowed(self, action):
        if action.type == "screenshot":
            return True
        if action.type != "click" or action.button != "left" or action.keys:
            return False
        box = await self.page.locator("summary").bounding_box()
        return bool(
            box
            and box["x"] <= action.x < box["x"] + box["width"]
            and box["y"] <= action.y < box["y"] + box["height"]
        )

    async def click(self, action):
        # [中文] 批准后页面可能发生变化,点击执行前再次核对并绑定原目标。
        # 审批后页面可能已变化,因此要重新定位点击目标。
        if not await self.allowed(action):
            raise RuntimeError("Pickup details moved or the click is not allowed.")
        await self.page.mouse.click(action.x, action.y)

    async def refuse(self, action):
        raise RuntimeError(f"This exercise does not allow {action.type}.")

    double_click = drag = keypress = move = scroll = type = wait = refuse


async def main():
    async with (
        async_playwright() as playwright,
        AsyncOpenAI(timeout=30, max_retries=0) as client,
    ):
        browser = await playwright.chromium.launch(env={}, chromium_sandbox=True)
        try:
            page = await browser.new_page(viewport={"width": 1024, "height": 768})
            await page.route("**/*", lambda route: route.abort())
            await page.set_content(
                "<title>Parcel pickup</title><main>"
                "<h1>Parcel pickup</h1><p>Order #1042 is ready.</p>"
                "<details><summary>Show pickup details</summary>"
                "<p>Locker 18. Pickup code: 6249.</p></details></main>"
            )
            driver = PickupDriver(page)
            tools = AsyncComputerToolset(driver)
            try:
                response = await tools.run(
                    client,
                    {
                        "model": "gpt-6.1-sol",
                        "store": False,
                        "input": "Take a screenshot. Open 'Show pickup details' and report the locker and pickup code. Use only screenshots and left clicks.",
                    },
                    before_action=driver.allowed,
                    max_steps=6,
                )
                if await page.locator("details").get_attribute("open") is None:
                    raise RuntimeError("Pickup details are still closed.")
                print(
                    "Verified pickup details:",
                    await page.locator("details p").inner_text(),
                )
                print("Answer:", response.output_text)
            finally:
                tools.close()
        finally:
            await browser.close()


if __name__ == "__main__":
    asyncio.run(main())

运行 .venv/bin/python pickup.py 或 node pickup.mjs。成功打印 Verified pickup details: Locker 18. Pickup code: 6249.(已核验取件详情:18号柜,取件码6249),随后显示模型回答。待处理API安全检查会使此示例停止;只有应用能处理时才添加人工批准回调。

浏览器控件示例。 模型在访客证表单填写姓名Mira并点击Create badge(创建访客证)。驱动只暴露页面读取、输入和点击;确认回调仅允许该表单控件及姓名Mira。脚本检查页面结果后再报告成功。保存为 badge.py 或 badge.mjs:

来源示例4(javascript)

程序中的自然语言译释43:

为Mira创建访客证。

程序中的自然语言译释44:

访客证未成功创建,后面附实际结果。

// [中文] 访客证驱动通过state和read_page报告控件,confirm仅允许Mira及指定控件,结束时验收页面。
import OpenAI from "openai";
import { BrowserToolset } from "openai/helpers/browser";
import { ResponsesToolRunner } from "openai/helpers/responses-tool-runner";
import { chromium } from "playwright";

class BadgeForm {
  constructor(page) {
    this.page = page;
  }

  // [中文] 读取当前标签状态;这里报告观察结果,不授予操作权限。
  async state() {
    return {
      tabs: [
        {
          id: "badge",
          title: await this.page.title(),
          url: this.page.url(),
          active: true,
        },
      ],
    };
  }

  // [中文] 读取页面文字并返回结构化控件引用,供后续定位。
  async read_page(_args) {
    return {
      version: 1,
      text: await this.page.locator("body").innerText(),
      elements: [
        { ref: "name", role: "textbox", label: "Name" },
        { ref: "save", role: "button", label: "Create badge" },
      ],
      omitted: [],
    };
  }

  async type(args) {
    await this.page.locator("#name").fill(args.text);
  }

  async click(_args) {
    await this.page.locator("#save").click();
  }

  // [中文] 按当前页面及固定目标参数判断是否允许;只有明确批准才继续。
  async confirm({ member, arguments: args }) {
    if (
      this.page.url() !== "about:blank" ||
      (args.tab_id !== null && args.tab_id !== "badge")
    ) {
      return false;
    }
    return (
      (member === "type" && args.target === "name" && args.text === "Mira") ||
      (member === "click" && args.target === "save")
    );
  }
}

async function main() {
  const client = new OpenAI({ timeout: 30_000, maxRetries: 0 });
  const browser = await chromium.launch({ env: {}, chromiumSandbox: true });
  try {
    const page = await browser.newPage();
    await page.route("**/*", (route) => route.abort());
    await page.setContent(`
      <title>Visitor badge</title>
      <label>Name <input id="name"></label>
      <button id="save" onclick="document.querySelector('output').innerText =
        'Badge ready: ' + document.querySelector('#name').value">Create badge</button>
      <output></output>
    `);
    const driver = new BadgeForm(page);
    const tools = new BrowserToolset(driver, {
      members: ["read_page", "type", "click"],
      confirm: driver.confirm.bind(driver),
    });
    const response = await new ResponsesToolRunner({ browser: tools }).run(
      client,
      {
        model: "gpt-6-astra",
        store: false,
        input: "Create a visitor badge for Mira.",
      },
      { maxSteps: 6 }
    );
    const result = await page.locator("output").innerText();
    if (result !== "Badge ready: Mira") {
      throw new Error(`The badge was not created: ${JSON.stringify(result)}`);
    }
    console.log(result);
    console.log("Answer:", response.output_text);
  } finally {
    await browser.close();
  }
}

await main();

来源示例5(python)

本段复用的中文释义:译释43、译释44。

## [中文] 访客证驱动通过state和read_page报告控件,confirm仅允许Mira及指定控件,结束时验收页面。
import asyncio

from openai import AsyncOpenAI
from openai.lib.browser import AsyncBrowserToolset
from openai.lib.responses_tool_runner import AsyncResponsesToolRunner
from playwright.async_api import async_playwright


class BadgeForm:
    def __init__(self, page):
        self.page = page

    # [中文] 读取当前标签状态;这里报告观察结果,不授予操作权限。
    async def state(self):
        return {
            "tabs": [
                {
                    "id": "badge",
                    "title": await self.page.title(),
                    "url": self.page.url,
                    "active": True,
                }
            ]
        }

    # [中文] 读取页面文字并返回结构化控件引用,供后续定位。
    async def read_page(self, args):
        return {
            "version": 1,
            "text": await self.page.locator("body").inner_text(),
            "elements": [
                {"ref": "name", "role": "textbox", "label": "Name"},
                {"ref": "save", "role": "button", "label": "Create badge"},
            ],
            "omitted": [],
        }

    async def type(self, args):
        await self.page.locator("#name").fill(args["text"])

    async def click(self, args):
        await self.page.locator("#save").click()

    # [中文] 按当前页面及固定目标参数判断是否允许;只有明确批准才继续。
    async def confirm(self, context):
        args = context["arguments"]
        if self.page.url != "about:blank" or args["tab_id"] not in (None, "badge"):
            return False
        return (
            context["member"] == "type"
            and args["target"] == "name"
            and args["text"] == "Mira"
        ) or (context["member"] == "click" and args["target"] == "save")


async def main():
    async with (
        async_playwright() as playwright,
        AsyncOpenAI(timeout=30, max_retries=0) as client,
    ):
        browser = await playwright.chromium.launch(env={}, chromium_sandbox=True)
        try:
            page = await browser.new_page()
            await page.route("**/*", lambda route: route.abort())
            await page.set_content("""
                <title>Visitor badge</title>
                <label>Name <input id="name"></label>
                <button id="save" onclick="document.querySelector('output').innerText =
                  'Badge ready: ' + document.querySelector('#name').value">Create badge</button>
                <output></output>
            """)
            driver = BadgeForm(page)
            tools = AsyncBrowserToolset(
                driver, members=["read_page", "type", "click"], confirm=driver.confirm
            )
            response = await AsyncResponsesToolRunner(browser=tools).run(
                client,
                {
                    "model": "gpt-6-astra",
                    "store": False,
                    "input": "Create a visitor badge for Mira.",
                },
                max_steps=6,
            )
            result = await page.locator("output").inner_text()
            if result != "Badge ready: Mira":
                raise RuntimeError(f"The badge was not created: {result!r}")
            print(result)
            print("Answer:", response.output_text)
        finally:
            await browser.close()


if __name__ == "__main__":
    asyncio.run(main())

运行相应脚本,成功输出 Badge ready: Mira(访客证已就绪:Mira)及模型回答。API拒绝工具或Chromium无法启动时参见运行排错,点击偏移时参见截图与坐标。

检查模型权限、成本与数据

  • 模型:浏览器函数工具要求支持Responses和函数调用,返回图片时还需图像输入;截图式操作须核对模型Tools栏的computer能力及项目访问权。
  • 成本:每次请求可产生输入输出词元费用,截图计为图像输入;核对价格、图像成本及请求用量。托管环境可能另收费。
  • 数据:SDK会将返回的页面文字与截图发给OpenAI。store: false 不取消所有数据保留;核对端点数据控制、图像输入例外及环境提供方政策。

查看并批准变更

可尝试Python预约演示或Node演示:填写虚构预约,提交前询问批准,检查保存字段和确认结果。用户可观看、接管、交回或停止。两者均提供真实浏览器中的无API本地演练。观察桌面SDK操作可用可视桌面演示。

调整驱动

将示例页面换成自己控制的环境。浏览器用 members 只暴露任务所需操作;桌面实现必需方法并拒绝不支持操作。运行结束前保持会话,之后关闭应用创建的资源。

页面内容会影响模型,应使用新浏览器配置或隔离桌面,限制网络,避免凭证出现在文字与截图中。变更前审批,报告成功前检查实际保存结果。驱动参考说明方法协议、审批、URL策略和恢复。

对有界选择切换到Decisions

模型需选择动作并生成参数时使用Responses。应用已知所有允许动作及完整参数时才用Decisions,例如预先定义 route_billing、route_technical、review。Decisions只选择,应用仍需批准、执行并验收;页面变化后应丢弃旧选项。参见Decisions。

在预约演示中,用有Decisions权限的密钥,设置 OPENAI_DECISIONS_MODEL 并重启服务器,开始新运行再选择Decisions。

直接使用API

需要自行管理工具循环时直接用Responses。应用提供环境、执行模型请求。可选:模型通过PyAutoGUI或Playwright生成操作代码,单次调用组合动作、循环与条件;或使用computer工具,接收结构化鼠标键盘操作。GPT-6 Astra推荐代码执行方式,computer仍可用。

已有函数调用或MCP界面工具也可保留,差异见自有UI工具。

使用代码执行

给模型一个接受脚本的函数工具,应用在隔离浏览器或桌面执行,返回文本及截图。调用之间保持环境,使模型能延续工作。

运行示例应用

CUA示例应用有JavaScript/Playwright与Python/PyAutoGUI实现、内置任务和共用控制台。按选定实现的说明在隔离环境安装;选择场景运行;检查动作、截图和最终状态。安装、桌面权限与支持环境以README为准,接入真实账户前先阅读安全要求。

连接自己的运行时

下例Python、Ruby、Go向PyAutoGUI桌面运行时发送Python代码,JavaScript用Playwright操作浏览器。均使用普通函数工具,以原始 call_id 返回文本或图像。

execute_in_sandbox 或 executeInSandbox 由应用实现,负责保持会话、限制执行及权限;它们是集成片段,不是直接运行示例应用的代码。

来源示例6(python)

程序中的自然语言译释45:

在持久桌面中运行Python,变量跨调用保留;PyAutoGUI操作为同步。
可用对象包括pyautogui、time、log(value)、display(PIL_image)。
操作前用display(pyautogui.screenshot())检查屏幕。使用截图坐标,执行少量动作后再检查。
截图保留在内存,并保持PyAutoGUI的fail-safe开启。

程序中的自然语言译释46:

任务达到20个响应的上限,请检查最后结果。

## [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
import json
import uuid

from openai import OpenAI
from openai.types.responses import (
    FunctionToolParam,
    ResponseInputParam,
)

def run_computer_use(endpoint, prompt, model="gpt-6-astra"):
    client = OpenAI()
    session_id = str(uuid.uuid4())
    tools: list[FunctionToolParam] = [
        {
            "type": "function",
            "name": "exec_py",
            "description": (
                "Run Python in a persistent desktop. Variables persist across calls. "
                "PyAutoGUI operations are synchronous. Available: pyautogui, time, "
                "log(value), and display(PIL_image). Inspect the screen with "
                "display(pyautogui.screenshot()) before acting. Use screenshot "
                "coordinates and check the screen after a short group of actions. "
                "Keep screenshots in memory and PyAutoGUI's fail-safe enabled."
            ),
            "parameters": {
                "type": "object",
                "properties": {"code": {"type": "string"}},
                "required": ["code"],
                "additionalProperties": False,
            },
            "strict": True,
        }
    ]
    next_input: ResponseInputParam = [{"role": "user", "content": prompt}]
    previous_response_id = None

    for turn in range(20):
        response = client.responses.create(
            model=model,
            tools=tools,
            input=next_input,
            previous_response_id=previous_response_id,
        )
        if response.status != "completed":
            raise RuntimeError(f"Response stopped with status: {response.status}")

        calls = [item for item in response.output if item.type == "function_call"]
        if not calls and any(
            item.type == "message" and item.phase != "commentary"
            for item in response.output
        ):
            print(response.output_text)
            return
        if turn == 19:
            raise RuntimeError(
                "The task reached the 20-response limit. Inspect the last result."
            )

        next_input = []
        for call in calls:
            if call.name != "exec_py":
                raise ValueError(f"Unexpected tool: {call.name}")
            code = json.loads(call.arguments)["code"]
            output = execute_in_sandbox(code, session_id, endpoint)
            next_input.append(
                {
                    "type": "function_call_output",
                    "call_id": call.call_id,
                    "output": output,
                }
            )
        previous_response_id = response.id

来源示例7(javascript)

程序中的自然语言译释107:

在持久浏览器中运行JavaScript。可用Playwright的browser、context、page对象,以及console.log(value)和display(base64Image)。
把可复用变量保存在globalThis。操作前查看截图,少量动作后再检查屏幕,截图保留在内存。
异步操作使用顶层await;display()返回图像,console.log()返回简洁文字。上下文视口为1440×900。

本段复用的中文释义:译释46。

// [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
import { randomUUID } from "node:crypto";
import OpenAI from "openai";

async function runComputerUse(endpoint, prompt, model = "gpt-6-astra") {
  const client = new OpenAI();
  const sessionId = randomUUID();

  const tools = [
    {
      type: "function",
      name: "exec_js",
      description: `Run JavaScript in a persistent browser. Available: Playwright's
browser, context, and page objects; console.log(value); and display(base64Image).
Save reusable variables on globalThis. Inspect a screenshot before acting and
check the screen after a short group of actions. Keep screenshots in memory.
Use top-level await for async operations. Return images with display() and concise
text with console.log(). The context viewport is 1440x900.`,
      parameters: {
        type: "object",
        properties: { code: { type: "string" } },
        required: ["code"],
        additionalProperties: false,
      },
      strict: true,
    },
  ];

  let nextInput = [{ role: "user", content: prompt }];
  let previousResponseId;

  for (let turn = 0; turn < 20; turn++) {
    const response = await client.responses.create({
      model,
      tools,
      input: nextInput,
      previous_response_id: previousResponseId,
      reasoning: { effort: "low" },
    });
    // [中文] 响应未完整完成时停止,避免执行部分生成的请求。
    if (response.status !== "completed") {
      throw new Error(`Response stopped with status: ${response.status}`);
    }
    const calls = response.output.filter(
      (item) => item.type === "function_call"
    );
    if (
      calls.length === 0 &&
      response.output.some(
        (item) => item.type === "message" && item.phase !== "commentary"
      )
    ) {
      console.log(response.output_text);
      return;
    }
    if (turn === 19) {
      throw new Error(
        "The task reached the 20-response limit. Inspect the last result."
      );
    }

    nextInput = [];
    for (const call of calls) {
      if (call.name !== "exec_js")
        throw new Error(`Unexpected tool: ${call.name}`);
      const { code } = JSON.parse(call.arguments);
      const output = await executeInSandbox(code, sessionId, endpoint);
      nextInput.push({
        type: "function_call_output",
        call_id: call.call_id,
        output,
      });
    }
    previousResponseId = response.id;
  }
}

来源示例8(ruby)

本段复用的中文释义:译释45。

## [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
require "json"
require "openai"
require "securerandom"

def run_computer_use(endpoint, prompt)
  client = OpenAI::Client.new
  session_id = SecureRandom.uuid
  tools = [
    {
      type: :function,
      name: "exec_py",
      description: "Run Python in a persistent desktop. Variables persist across calls. PyAutoGUI operations are synchronous. Available: pyautogui, time, log(value), and display(PIL_image). Inspect the screen with display(pyautogui.screenshot()) before acting. Use screenshot coordinates and check the screen after a short group of actions. Keep screenshots in memory and PyAutoGUI's fail-safe enabled.",
      parameters: {
        type: :object,
        properties: { code: { type: :string } },
        required: ["code"],
        additionalProperties: false
      },
      strict: true
    }
  ]
  next_input = []
  next_input << {
    role: :user,
    content: prompt
  }
  history = {}
  20.times do |turn|
    response = client.responses.create(
      model: "gpt-6-astra", tools: tools, input: next_input, previous_response_id: history[:id]
    )
    raise "Response stopped with status: #{response.status}" unless response.status == OpenAI::Responses::ResponseStatus::COMPLETED

    calls = response.output.grep(OpenAI::Responses::ResponseFunctionToolCall)
    if calls.empty? && response.output.any? { |item| item.is_a?(OpenAI::Responses::ResponseOutputMessage) && item.phase != :commentary }
      puts(response.output_text)
      return response
    end
    raise "The task reached the 20-response limit" if turn == 19

    next_input.clear
    calls.each do |call|
      raise "Unexpected tool: #{call.name}" unless call.name == "exec_py"

      code = JSON.parse(call.arguments).fetch("code")
      raise "Expected Python source text" unless code.is_a?(String)

      output = execute_in_sandbox(code, session_id, endpoint)
      next_input << {
        type: :function_call_output,
        call_id: call.call_id,
        output: output
      }
    end
    history[:id] = response.id
  end
end

来源示例9(go)

本段复用的中文释义:译释45。

// [中文] 通过普通函数把生成脚本交给应用的隔离执行服务;会话与previous_response_id分别保持环境和模型上下文。
func runComputerUse(ctx context.Context, client openai.Client, endpoint, prompt string, terminal *bufio.Scanner, output io.Writer) error {
    session := make([]byte, 16)
    if _, err := rand.Read(session); err != nil {
        return err
    }
    sessionID := hex.EncodeToString(session)
    tool := responses.ToolParamOfFunction("exec_py", map[string]any{
        "type": "object", "properties": map[string]any{"code": map[string]any{"type": "string"}},
        "required": []string{"code"}, "additionalProperties": false,
    }, true)
    tool.OfFunction.Description = openai.String("Run Python in a persistent desktop. Variables persist across calls. " +
        "PyAutoGUI operations are synchronous. Available: pyautogui, time, " +
        "log(value), and display(PIL_image). Inspect the screen with " +
        "display(pyautogui.screenshot()) before acting. Use screenshot " +
        "coordinates and check the screen after a short group of actions. " +
        "Keep screenshots in memory and PyAutoGUI's fail-safe enabled.")
    params := responses.ResponseNewParams{
        Model: "gpt-6-astra", Tools: []responses.ToolUnionParam{tool},
        Input: responses.ResponseNewParamsInputUnion{OfString: openai.String(prompt)},
    }
    for turn := 0; turn < 20; turn++ {
        response, err := client.Responses.New(ctx, params)
        if err != nil {
            return err
        }
        if response.Status != "completed" {
            return fmt.Errorf("response stopped with status: %s", response.Status)
        }
        var calls []responses.ResponseFunctionToolCall
        finalMessage := false
        for _, item := range response.Output {
            if item.Type == "function_call" {
                calls = append(calls, item.AsFunctionCall())
            }
            if item.Type == "message" && item.AsMessage().Phase != "commentary" {
                finalMessage = true
            }
        }
        if len(calls) == 0 && finalMessage {
            fmt.Fprintln(output, response.OutputText())
            return nil
        }
        if turn == 19 {
            return fmt.Errorf("task reached the 20-response limit; inspect the last result")
        }
        nextInput := responses.ResponseInputParam{}
        for _, call := range calls {
            if call.Name != "exec_py" {
                return fmt.Errorf("unexpected tool: %s", call.Name)
            }
            var arguments struct {
                Code *string `json:"code"`
            }
            if err := json.Unmarshal([]byte(call.Arguments), &arguments); err != nil {
                return err
            }
            if arguments.Code == nil {
                return fmt.Errorf("exec_py requires code")
            }
            observations, err := executeInSandbox(ctx, endpoint, sessionID, *arguments.Code, terminal, output)
            if err != nil {
                return err
            }
            nextInput = append(nextInput, responses.ResponseInputItemUnionParam{
                OfFunctionCallOutput: &responses.ResponseInputItemFunctionCallOutputParam{
                    CallID: openai.String(call.CallID),
                    Output: responses.ResponseInputItemFunctionCallOutputOutputUnionParam{OfResponseFunctionCallOutputItemArray: observations},
                },
            })
        }
        params.Input = responses.ResponseNewParamsInputUnion{OfInputItemList: nextInput}
        params.PreviousResponseID = openai.String(response.ID)
    }
    return nil
}

完整适配器及输出形态见连接执行服务。该服务属于你的应用,不是OpenAI托管端点。

保持状态并返回观察结果

保持浏览器或桌面会话;持久Python或JavaScript命名空间也可保留变量。工具定义要描述可用对象与辅助函数。界面未知时先返回当前截图,执行少量动作后再截图核对。图像保存在内存,采用 detail: original。缩图后应把模型坐标映射回环境坐标,参见截图。

API会话与执行环境是两套状态。会话保留调用和结果,应用保留对应环境;续接响应不会恢复浏览器、登录或运行时变量。

使用computer工具

需要结构化动作而非生成代码时采用。按同一示例应用设置选择Native模式,并选支持computer的模型。流程是发送任务、执行动作、返回截图。示例页面有Show filters(显示筛选)和搜索框,可按自己界面调整。

环境与动作处理器见集成方案。

发送任务

在 tools 启用 computer,说明期望结果:

来源示例10(javascript)

程序中的自然语言译释47:

检查Filters(筛选)面板是否打开;未打开则点击Show filters。随后在搜索框输入penguin。使用computer工具进行界面交互。

程序中的自然语言译释69:

检查Filters面板是否打开。

// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
import OpenAI from "openai";

const client = new OpenAI();

const response = await client.responses.create({
  model: "gpt-6.1-sol",
  tools: [{ type: "computer" }],
  input:
    "Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",
});

console.log(JSON.stringify(response.output, null, 2));

来源示例11(python)

本段复用的中文释义:译释47、译释69。

## [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-6.1-sol",
    tools=[{"type": "computer"}],
    input="Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.",
)

print(response.output)

来源示例12(go)

本段复用的中文释义:译释47、译释69。

// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
package main

import (
    "context"
    "fmt"

    "github.com/openai/openai-go/v3"
    "github.com/openai/openai-go/v3/responses"
)

func main() {
    client := openai.NewClient()
    response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{
        Model: "gpt-6.1-sol",
        Tools: []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},
        Input: responses.ResponseNewParamsInputUnion{OfString: openai.String("Check whether the Filters panel is open. If it is not open, click Show filters. Then type penguin in the search box. Use the computer tool for UI interaction.")},
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(response.Output)
}

来源示例13(java)

程序中的自然语言译释129:

必要时打开Filters面板,再搜索penguin;使用computer工具进行界面交互。

// [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.responses.ResponseCreateParams;
import java.util.List;
import java.util.Map;

ResponseCreateParams params =
    ResponseCreateParams.builder()
        .model("gpt-6.1-sol")
        .input(
            "Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.")
        .putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))
        .build();

client.responses().create(params).output().forEach(System.out::println);

来源示例14(ruby)

本段复用的中文释义:译释129。

## [中文] 启用computer工具并提交筛选和搜索任务;动作需要应用另行执行。
require "openai"

client = OpenAI::Client.new
response = client.responses.create(
  model: "gpt-6.1-sol",
  input: "Open the Filters panel if needed, then search for penguin. Use the computer tool for UI interaction.",
  tools: [{ type: :computer }]
)

puts(response.output)

执行请求动作

computer_call 含有序 actions。下例选择搜索框并输入 penguin:

来源示例15(json):以下保留协议载荷、配置、返回示意或语法定义;省略号及片段不代表完整可运行程序。

{
  "output": [
    {
      "type": "computer_call",
      "call_id": "call_002",
      "actions": [
        { "type": "click", "button": "left", "x": 405, "y": 157 },
        { "type": "type", "text": "penguin" }
      ],
      "status": "completed"
    }
  ]
}

处理器按序把允许的动作映射到浏览器或操作系统输入,再截取更新画面。动作包括 click、double_click、drag、move、scroll、keypress、type、wait、screenshot。

首次可能只有截图请求,此时不改变UI,只返回当前画面。调用 status: completed 仅表示模型生成该调用完毕,应用仍需执行。

键位、拖拽及修饰键映射见动作处理器。

返回截图

返回 call_id 对齐的 computer_call_output,使用 previous_response_id 续接模型会话。

来源示例16(javascript)

// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
import OpenAI from "openai";

const client = new OpenAI();

async function sendComputerScreenshot(response, callId, screenshotBase64) {
  const output = {
    type: "computer_screenshot",
    image_url: `data:image/png;base64,${screenshotBase64}`,
    detail: "original",
  };

  return await client.responses.create({
    model: "gpt-6.1-sol",
    tools: [{ type: "computer" }],
    previous_response_id: response.id,
    input: [
      {
        type: "computer_call_output",
        call_id: callId,
        output,
      },
    ],
  });
}

来源示例17(python)

## [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
from openai import OpenAI

client = OpenAI()


def send_computer_screenshot(response, call_id, screenshot_base64):
    return client.responses.create(
        model="gpt-6.1-sol",
        tools=[{"type": "computer"}],
        previous_response_id=response.id,
        input=[
            {
                "type": "computer_call_output",
                "call_id": call_id,
                "output": {
                    "type": "computer_screenshot",
                    "image_url": f"data:image/png;base64,{screenshot_base64}",
                    "detail": "original",
                },
            }
        ],
    )

来源示例18(go)

// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
package main

import (
    "context"
    "fmt"

    "github.com/openai/openai-go/v3"
    "github.com/openai/openai-go/v3/responses"
)

func main() {
    client := openai.NewClient()
    response, err := sendComputerScreenshot(client, "resp_abc123", "call_abc123", "<base64 bytes here>")
    if err != nil {
        panic(err)
    }
    fmt.Println(response.Output)
}

func sendComputerScreenshot(client openai.Client, responseID string, callID string, screenshotBase64 string) (*responses.Response, error) {
    screenshot := responses.ResponseComputerToolCallOutputScreenshotParam{
        ImageURL: openai.String("data:image/png;base64," + screenshotBase64),
    }
    screenshot.SetExtraFields(map[string]any{"detail": "original"})
    return client.Responses.New(context.Background(), responses.ResponseNewParams{
        Model:              "gpt-6.1-sol",
        Tools:              []responses.ToolUnionParam{{OfComputer: &responses.ComputerToolParam{}}},
        PreviousResponseID: openai.String(responseID),
        Input: responses.ResponseNewParamsInputUnion{OfInputItemList: responses.ResponseInputParam{
            responses.ResponseInputItemParamOfComputerCallOutput(callID, screenshot),
        }},
    })
}

来源示例19(java)

// [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.core.JsonValue;
import com.openai.models.responses.ResponseComputerToolCallOutputScreenshot;
import com.openai.models.responses.ResponseCreateParams;
import com.openai.models.responses.ResponseInputItem;
import java.util.List;
import java.util.Map;

String responseId = "resp_abc123";

String computerCallId = "call_abc123";

String screenshotBase64 = "<base64 bytes here>";

ResponseCreateParams params =
    ResponseCreateParams.builder()
        .model("gpt-6.1-sol")
        .input(
            ResponseCreateParams.Input.ofResponse(
                List.of(
                    ResponseInputItem.ofComputerCallOutput(
                        ResponseInputItem.ComputerCallOutput.builder()
                            .callId(computerCallId)
                            .output(
                                ResponseComputerToolCallOutputScreenshot.builder()
                                    .imageUrl("data:image/png;base64," + screenshotBase64)
                                    .putAdditionalProperty("detail", JsonValue.from("original"))
                                    .build())
                            .build()))))
        .previousResponseId(responseId)
        .putAdditionalBodyProperty("tools", JsonValue.from(List.of(Map.of("type", "computer"))))
        .build();

client.responses().create(params).output().forEach(System.out::println);

来源示例20(ruby)

## [中文] 以原call_id返回computer_call_output截图,并用previous_response_id续接。
require "openai"

client = OpenAI::Client.new
response = client.responses.create(
  model: "gpt-6.1-sol",
  previous_response_id: "resp_abc123",
  input: [
    {
      type: :computer_call_output,
      call_id: "call_abc123",
      output: {
        type: :computer_screenshot,
        image_url: "data:image/png;base64,<base64 bytes here>",
        detail: :original
      }
    }
  ],
  tools: [{ type: :computer }]
)

puts(response.output)

保持环境可用,遵循相同截图与状态规则。

重复到不再返回computer调用,检查其余输出是回答、求助还是其他工具,再在应用验收;本例应打开筛选面板,搜索框含 penguin。循环骨架见重复操作循环。

安全运行

  • 隔离浏览器或虚拟机,限制站点与操作到任务所需范围。
  • 页面、文档和工具结果均为不可信内容,不能赋予权限或覆盖用户指令。
  • 购买、传输数据、破坏性或难以撤销操作应由用户掌控;向表单输入敏感信息也属于传输。
  • 设置步数、时间或成本上限,支持取消,检查实际结果,而非只信最终回答。

具体审批、人工接管和指令例子参见确认与同意。

后续参考

集成方案介绍环境、处理器、截图和执行服务;旧预览迁移介绍旧集成升级;CUA示例应用提供完整浏览器与桌面工作流。

OpenAI 代码解释器
OpenAI 计算机操作集成方案
温馨提示
下载编程狮App,免费阅读超1000+编程语言教程
取消
确定
目录

其他能力与进阶指南

Embeddings 接口参考

审计日志接口参考

关闭

MIP.setData({ 'pageTheme' : getCookie('pageTheme') || {'day':true, 'night':false}, 'pageFontSize' : getCookie('pageFontSize') || 20 }); MIP.watch('pageTheme', function(newValue){ setCookie('pageTheme', JSON.stringify(newValue)) }); MIP.watch('pageFontSize', function(newValue){ setCookie('pageFontSize', newValue) }); function setCookie(name, value){ var days = 1; var exp = new Date(); exp.setTime(exp.getTime() + days*24*60*60*1000); document.cookie = name + '=' + value + ';expires=' + exp.toUTCString(); } function getCookie(name){ var reg = new RegExp('(^| )' + name + '=([^;]*)(;|$)'); return document.cookie.match(reg) ? JSON.parse(document.cookie.match(reg)[2]) : null; }