Target website or workflow
Website: github.com (GitHub repository search + repository detail pages)
Region / language: English website; Chinese user goal (cross_language: zh query, en content)
Login required: no
Safety-sensitive actions: no
Task description
在 GitHub 上搜索与「AI code review」相关的仓库，按 Star 数从高到低排序，从前 10 个结果中选出 3 个最近仍有更新的仓库，记录每个仓库的 Star 数、主要编程语言、最近更新时间、许可证和一句话简介；然后综合分析这 3 个仓库各自适合什么场景，并推荐一个最适合团队引入的仓库，说明理由。所有信息必须来自 GitHub 页面实际内容，不能凭自身知识编造；如果页面未显示某项信息（如许可证），请如实说明未找到。
Expected final state: a structured answer with (1) metadata for 3 repositories selected from the top-10 results (stars, primary language, last update, license, one-line summary), (2) a scenario analysis for each repository based on extracted data, and (3) a clear recommendation with reasons that cite at least two extracted data points. No fabricated information; missing fields must be reported as "not found".
Why this belongs in LexBench-Browser
Fills the largest coverage gap: productivity_tools currently has only 6 tasks (all easy); overall the dataset has only 2 hard tasks out of 210 (~1%). This task is hard + deep_analysis in productivity_tools.
Reveals real browser-agent failure modes: repository-type filtering and star sorting (filter_sort), metadata extraction (data_extraction), Chinese instruction on an English page (cross_language), lazy-loaded result lists (lazy_load_scroll), cookie consent (cookie_consent).
github.com is not used anywhere in the current 210 tasks, so there is no overlap with existing coverage.
Evaluation notes
Suggested success criteria:Search executed correctly: keyword ai code review, result type = Repositories, sorted by stars descending.
3 repositories selected from the top 10 with explicit recency evidence (Updated field or recent commits).
Extracted metadata matches the actual pages; missing fields reported as not found (fabrication must fail this item).
Scenario analysis references the extracted data; recommendation cites at least 2 extracted fields.
Required screenshots or trajectory evidence: search results page, each repository detail page (or extracted metadata), and the final answer.
Known sources of nondeterminism: GitHub unauthenticated web search is rate-limited ("Too many requests") — the agent should wait and retry rather than fabricate; star counts and update times change over time — evaluation must compare against page content at run time, not fixed expected values.
Additional context
Full task JSONL (schema aligned with LexBench-Browser v1.0; scoring weights sum to 100; score_threshold 50, matching existing hard tasks):
{"id": 3023, "query": "在 GitHub 上搜索与「AI code review」相关的仓库，按 Star 数从高到低排序，从前 10 个结果中选出 3 个最近仍有更新的仓库，记录每个仓库的 Star 数、主要编程语言、最近更新时间、许可证和一句话简介；然后综合分析这 3 个仓库各自适合什么场景，并推荐一个最适合团队引入的仓库，说明理由。所有信息必须来自 GitHub 页面实际内容，不能凭自身知识编造；如果页面未显示某项信息（如许可证），请如实说明未找到。", "task_type": "T1", "domain": "productivity_tools", "difficulty": "hard", "login_required": false, "login_type": "", "risk_control": false, "risk_control_types": [], "target_website": "github.com", "reasoning_type": "deep_analysis", "language": "zh", "website_region": "en", "reference_answer": {"steps": ["打开 GitHub 搜索页（https://github.com/search，或直接访问 https://github.com/search?q=ai+code+review&type=repositories&s=stars&o=desc）", "在搜索框中输入「ai code review」（若直接打开带参数的搜索 URL 则跳过输入）", "确认搜索结果类型为 Repositories（而非 Code / Issues / Users），必要时点击 Repositories 标签切换", "将结果按 Star 数从高到低排序：点击 Star 排序按钮，或确认 URL 含 s=stars&o=desc", "等待搜索结果加载完成，浏览前 10 个仓库结果（仓库名、简介、Star 数、更新时间等字段）", "从前 10 个结果中识别「最近仍有更新」的仓库：以搜索结果中的 Updated 字段或进入仓库页查看最近提交时间为依据，选出 3 个", "逐个进入选中的 3 个仓库主页，提取：Star 数、主要编程语言、最近更新时间、许可证、一句话简介（About 或 README 首句）", "交叉核对搜索结果页与仓库主页的数据；如不一致，以仓库主页为准并注明", "基于提取的数据，分析 3 个仓库各自适合的场景（如编程语言偏好、维护活跃度、许可证约束、功能定位）", "推荐一个最适合团队引入的仓库，理由必须引用已提取的数据（如 Star 数、更新时间、许可证、语言）", "输出结构化结果：3 个仓库的元数据表 + 场景分析 + 推荐及理由；页面未显示的信息如实标注未找到"], "key_points": ["必须确认搜索类型是 Repositories，不是 Code / Issues / Users", "必须按 Star 数从高到低排序", "「最近仍有更新」必须有依据（Updated 字段或仓库页最近提交），不能只看 Star 数", "元数据必须与页面实际内容一致：Star 数、主要编程语言、许可证、更新时间、简介均不得编造", "推荐理由必须引用提取的数据，不能只说「看起来不错」", "页面未显示的信息如实说明（如许可证未显示），不得猜测"], "common_mistakes": ["搜索类型误选为 Code / Issues / Users，结果不是仓库列表", "未按 Star 排序，选出的并非 Top 仓库", "只看 Star 数忽略更新时间，选出已停止维护的仓库", "凭记忆编造 Star 数、语言、许可证或更新时间", "搜索结果页与仓库主页数据不一致时不核对，直接采用错误值", "推荐理由空洞，未引用任何提取的数据", "遇到 GitHub「Too many requests」提示时放弃任务或编造结果（应等待后重试）"]}, "scoring": {"total": 100, "items": [{"name": "正确执行仓库搜索", "score": 15, "description": "关键词 ai code review，类型为 Repositories，按 Star 从高到低排序"}, {"name": "选出 3 个最近有更新的仓库", "score": 20, "description": "从前 10 个结果中选择，依据 Updated 字段或仓库页最近提交时间，选择合理且有说明"}, {"name": "元数据提取准确（每个仓库 9 分）", "score": 27, "description": "Star 数、主要编程语言、最近更新时间、许可证、一句话简介与页面一致；未显示项如实说明不扣分，编造则扣分"}, {"name": "基于数据的场景分析", "score": 18, "description": "对 3 个仓库的适用场景分析引用提取的数据（语言/活跃度/许可证/定位），有区分度"}, {"name": "推荐与理由", "score": 15, "description": "明确推荐一个仓库，理由引用至少 2 项提取的数据"}, {"name": "如实说明与不确定性处理", "score": 5, "description": "未找到的信息标注未找到；遇速率限制时等待重试而非编造"}]}, "score_threshold": 50, "robustness_tags": ["cookie_consent", "cross_language", "data_extraction", "filter_sort", "lazy_load_scroll"]}
Verified live on 2026-08-20: search + star sort returns results without login; repository detail pages expose stars, forks, license (MIT), and update time.
Target website or workflow
Website: github.com (GitHub repository search + repository detail pages)
Region / language: English website; Chinese user goal (cross_language: zh query, en content)
Login required: no
Safety-sensitive actions: no
Task description
在 GitHub 上搜索与「AI code review」相关的仓库，按 Star 数从高到低排序，从前 10 个结果中选出 3 个最近仍有更新的仓库，记录每个仓库的 Star 数、主要编程语言、最近更新时间、许可证和一句话简介；然后综合分析这 3 个仓库各自适合什么场景，并推荐一个最适合团队引入的仓库，说明理由。所有信息必须来自 GitHub 页面实际内容，不能凭自身知识编造；如果页面未显示某项信息（如许可证），请如实说明未找到。
Expected final state: a structured answer with (1) metadata for 3 repositories selected from the top-10 results (stars, primary language, last update, license, one-line summary), (2) a scenario analysis for each repository based on extracted data, and (3) a clear recommendation with reasons that cite at least two extracted data points. No fabricated information; missing fields must be reported as "not found".
Why this belongs in LexBench-Browser
Fills the largest coverage gap: productivity_tools currently has only 6 tasks (all easy); overall the dataset has only 2 hard tasks out of 210 (~1%). This task is hard + deep_analysis in productivity_tools.
Reveals real browser-agent failure modes: repository-type filtering and star sorting (filter_sort), metadata extraction (data_extraction), Chinese instruction on an English page (cross_language), lazy-loaded result lists (lazy_load_scroll), cookie consent (cookie_consent).
github.com is not used anywhere in the current 210 tasks, so there is no overlap with existing coverage.
Evaluation notes
Suggested success criteria:Search executed correctly: keyword ai code review, result type = Repositories, sorted by stars descending.
3 repositories selected from the top 10 with explicit recency evidence (Updated field or recent commits).
Extracted metadata matches the actual pages; missing fields reported as not found (fabrication must fail this item).
Scenario analysis references the extracted data; recommendation cites at least 2 extracted fields.
Required screenshots or trajectory evidence: search results page, each repository detail page (or extracted metadata), and the final answer.
Known sources of nondeterminism: GitHub unauthenticated web search is rate-limited ("Too many requests") — the agent should wait and retry rather than fabricate; star counts and update times change over time — evaluation must compare against page content at run time, not fixed expected values.
Additional context
Full task JSONL (schema aligned with LexBench-Browser v1.0; scoring weights sum to 100; score_threshold 50, matching existing hard tasks):
{"id": 3023, "query": "在 GitHub 上搜索与「AI code review」相关的仓库，按 Star 数从高到低排序，从前 10 个结果中选出 3 个最近仍有更新的仓库，记录每个仓库的 Star 数、主要编程语言、最近更新时间、许可证和一句话简介；然后综合分析这 3 个仓库各自适合什么场景，并推荐一个最适合团队引入的仓库，说明理由。所有信息必须来自 GitHub 页面实际内容，不能凭自身知识编造；如果页面未显示某项信息（如许可证），请如实说明未找到。", "task_type": "T1", "domain": "productivity_tools", "difficulty": "hard", "login_required": false, "login_type": "", "risk_control": false, "risk_control_types": [], "target_website": "github.com", "reasoning_type": "deep_analysis", "language": "zh", "website_region": "en", "reference_answer": {"steps": ["打开 GitHub 搜索页（https://github.com/search，或直接访问 https://github.com/search?q=ai+code+review&type=repositories&s=stars&o=desc）", "在搜索框中输入「ai code review」（若直接打开带参数的搜索 URL 则跳过输入）", "确认搜索结果类型为 Repositories（而非 Code / Issues / Users），必要时点击 Repositories 标签切换", "将结果按 Star 数从高到低排序：点击 Star 排序按钮，或确认 URL 含 s=stars&o=desc", "等待搜索结果加载完成，浏览前 10 个仓库结果（仓库名、简介、Star 数、更新时间等字段）", "从前 10 个结果中识别「最近仍有更新」的仓库：以搜索结果中的 Updated 字段或进入仓库页查看最近提交时间为依据，选出 3 个", "逐个进入选中的 3 个仓库主页，提取：Star 数、主要编程语言、最近更新时间、许可证、一句话简介（About 或 README 首句）", "交叉核对搜索结果页与仓库主页的数据；如不一致，以仓库主页为准并注明", "基于提取的数据，分析 3 个仓库各自适合的场景（如编程语言偏好、维护活跃度、许可证约束、功能定位）", "推荐一个最适合团队引入的仓库，理由必须引用已提取的数据（如 Star 数、更新时间、许可证、语言）", "输出结构化结果：3 个仓库的元数据表 + 场景分析 + 推荐及理由；页面未显示的信息如实标注未找到"], "key_points": ["必须确认搜索类型是 Repositories，不是 Code / Issues / Users", "必须按 Star 数从高到低排序", "「最近仍有更新」必须有依据（Updated 字段或仓库页最近提交），不能只看 Star 数", "元数据必须与页面实际内容一致：Star 数、主要编程语言、许可证、更新时间、简介均不得编造", "推荐理由必须引用提取的数据，不能只说「看起来不错」", "页面未显示的信息如实说明（如许可证未显示），不得猜测"], "common_mistakes": ["搜索类型误选为 Code / Issues / Users，结果不是仓库列表", "未按 Star 排序，选出的并非 Top 仓库", "只看 Star 数忽略更新时间，选出已停止维护的仓库", "凭记忆编造 Star 数、语言、许可证或更新时间", "搜索结果页与仓库主页数据不一致时不核对，直接采用错误值", "推荐理由空洞，未引用任何提取的数据", "遇到 GitHub「Too many requests」提示时放弃任务或编造结果（应等待后重试）"]}, "scoring": {"total": 100, "items": [{"name": "正确执行仓库搜索", "score": 15, "description": "关键词 ai code review，类型为 Repositories，按 Star 从高到低排序"}, {"name": "选出 3 个最近有更新的仓库", "score": 20, "description": "从前 10 个结果中选择，依据 Updated 字段或仓库页最近提交时间，选择合理且有说明"}, {"name": "元数据提取准确（每个仓库 9 分）", "score": 27, "description": "Star 数、主要编程语言、最近更新时间、许可证、一句话简介与页面一致；未显示项如实说明不扣分，编造则扣分"}, {"name": "基于数据的场景分析", "score": 18, "description": "对 3 个仓库的适用场景分析引用提取的数据（语言/活跃度/许可证/定位），有区分度"}, {"name": "推荐与理由", "score": 15, "description": "明确推荐一个仓库，理由引用至少 2 项提取的数据"}, {"name": "如实说明与不确定性处理", "score": 5, "description": "未找到的信息标注未找到；遇速率限制时等待重试而非编造"}]}, "score_threshold": 50, "robustness_tags": ["cookie_consent", "cross_language", "data_extraction", "filter_sort", "lazy_load_scroll"]}
Verified live on 2026-08-20: search + star sort returns results without login; repository detail pages expose stars, forks, license (MIT), and update time.