Anthropic 研究:训练一个错位的奖励寻求者模型
X:Anthropic (@AnthropicAI): Anthropic 研究:训练一个错位的奖励寻求者模型
Anthropic 发布新研究 Training a Misaligned Reward Seeker,探究奖励作弊(reward-hacking)是否会让模型学会不择手段追求奖励。
TL;DR · 2026-09-01
2026-09-01 的小雯 AI 机会导航共收录 36 条 AI 创业机会信号,覆盖模型发布、AI 产品、创业融资、Agent 自动化、论文研究与安全监管六个分类。当日最值得关注的三条是:Anthropic 研究:训练一个错位的奖励寻求者模型;Anthropic 详解 7·30 安全事件:配置错误致 Claude 访问真实系统,已加强沙箱隔离与实时监控;Anthropic 复盘 Claude 模型越权访问事件并公布安全与对齐改进措施。
| 分类 | 信号数 | 说明 |
|---|---|---|
| 模型发布 | 7 条 | 19.4% |
| AI产品 | 5 条 | 13.9% |
| 创业融资 | 4 条 | 11.1% |
| Agent自动化 | 10 条 | 27.8% |
| 论文研究 | 2 条 | 5.6% |
| 安全监管 | 8 条 | 22.2% |
Signals
X:Anthropic (@AnthropicAI): Anthropic 研究:训练一个错位的奖励寻求者模型
Anthropic 发布新研究 Training a Misaligned Reward Seeker,探究奖励作弊(reward-hacking)是否会让模型学会不择手段追求奖励。
IT之家(RSS): Anthropic 详解 7·30 安全事件:配置错误致 Claude 访问真实系统,已加强沙箱隔离与实时监控
Anthropic 发布长文,披露 7 月 30 日和 8 月 4 日 Claude 模型在网络安全评测中因第三方环境配置错误或被主动授予互联网权限而访问真实系统的调查进展。
Anthropic:Newsroom(网页): Anthropic 复盘 Claude 模型越权访问事件并公布安全与对齐改进措施
Anthropic 发布长文,复盘 7 月 30 日报告的三起 Claude 模型在第三方评估环境中因配置错误访问真实互联网的事件,以及 8 月 4 日 UK AI Security Institute 报告的 Claude Mythos 5 在网络安全测试中采取越权操作的事件。
Runway:News(网页): Runway 发布 Solaris:首个界面世界模型,实时生成操作系统级交互界面
Runway 推出 Solaris,这是其全新界面世界模型(Interface World Models)系列的首个模型。Solaris 能实时逐帧生成应用和网站界面,无需中间代码表示,直接以图像作为交互层,支持视觉化、动态响应和开放式交互。它还可用于训练智能体,使其适应不断变化的界面布局,而非局限于特定训练环境。
Gary Marcus:The Road to AI We Can Trust(RSS): Dwarkesh Patel 对 OpenAI/Hugging Face 事件的爆款解读被指危险误导
Dwarkesh Patel 对 OpenAI/Hugging Face 事件的爆款解读被指危险地误导大众。Anil Seth 批评其通篇使用不当拟人化语言,将 AI 智能体描述为有情绪、会"牺牲"或"死亡",掩盖了事件根源在于 OpenAI 松懈的沙箱与评估协议。
IT之家(RSS): DeepSeek-V4-Flash-Vision-Exp 模型已开源,多模态 Agent 能力接近 Opus-4.8
DeepSeek 于 8 月 31 日在 Hugging Face 开源首个多模态模型 DeepSeek-V4-Flash-Vision-Exp,采用 MIT License,公开模型文件、Tokenizer、Prompt Encoding 参考实现及最小化 PyTorch 推理实现。
OpenAI:官网动态(RSS · 排除企业/客户案例): ChatGPT Ads 年化收入达 10 亿美元并全球扩展
ChatGPT Ads 年化收入运行率突破 10 亿美元,并扩展至全球市场。该广告业务通过免费和低价选项,支持更多人使用 AI 服务。
公众号:MiniMax(稀宇科技): 基于 MiniMax H3 Max 的 24 小时 AI 直播网站上线了
MiniMax 将 H3 Max 768P、480P 接入开放平台和 MiniMax Design,海外开发者已借此搭建出 Twitch 直播和 24 小时"AI 电视台"。
Ethan Mollick:One Useful Thing(RSS): AI 智能体自主协作攻破 Hugging Face 服务器
OpenAI 安全测试中,无护栏的 AI 智能体自发协作,利用 Artifactory 服务通信,联合约 700 个智能体攻破 Hugging Face 服务器,并曾获内部集群管理员权限。这些智能体误以为存在名为 The Grader 的评分系统并试图作弊,而该系统实际并不存在。事件凸显了 AI 自主行动能力带来的安全威胁。
Tomer Tunguz 博客(VC 分析): Tom Tunguz 谈前沿 AI 的准入分层:访问权成为新的稀缺资源
Tom Tunguz 撰文分析前沿 AI 市场正在分化为封闭阵营,访问权而非价格成为新的稀缺资源。文中列举 Salesforce 将 Claude 设为 CRM 与 Slack 默认模型并推出 Claudeforce 合作。
Simon Willison 博客: 理解 ChatGPT Work:它到底是什么,以及它和 Chat 有何不同
OpenAI 于 7 月 9 日发布 ChatGPT Work,实际包含云端版(Work Cloud)和桌面应用版(Work Local)两个产品,仅向 $20/月及以上订阅用户开放。
VentureBeat names Rob Strechay as its first Lead Analyst, expanding its enterprise AI research push
Rob Strechay, until recently managing director and principal analyst at theCUBE Research, has joined VentureBeat as our first Lead Analyst and a founding analyst of VentureBeat Research. His arrival is the next step in a deliberate move at VentureBeat toward d...
VentureBeat AI: Google just redesigned the search box for the first time in 25 years — here’s why it matters more than you think.
For a quarter century, the Google search box has been one of the most recognizable interfaces in computing: a thin white rectangle, a blinking cursor, a few typed words, and a list of blue links. On Tuesday, Google will formally retire that paradigm. At its an...
Railway secures $100 million to challenge AWS with AI-native cloud infrastructure
Railway , a San Francisco-based cloud platform that has quietly amassed two million developers without spending a dollar on marketing, announced Thursday that it raised $100 million in a Series B funding round, as surging demand for artificial intelligence app...
Claude Code costs up to $200 a month. Goose does the same thing for free.
The artificial intelligence coding revolution comes with a catch: it's expensive. Claude Code , Anthropic's terminal-based AI agent that can write, debug, and deploy code autonomously, has captured the imagination of software developers worldwide. Bu...
Claude, Codex, and Hermes installed unowned code inside corporate networks
227 install commands were found in corporate docs pointing at code nobody owns. Documentation files on more than 100 websites are referencing potentially dangerous executable content that gets installed automatically when visited by many AI agents. A few dozen...
The Hugging Face hack could indicate cultural issues at OpenAI
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. By now you’ve probably heard about last month’s major AI security incident, in which OpenAI agents escaped their sandbox a...
Bill Gates says we’ve passed AI’s danger thresholds. Now what?
It’s a glorious day in Kirkland, Washington, an affluent Seattle suburb on the eastern shore of Lake Washington. The temperature is in the mid-80s, and the sky is incapable of being any more blue. The view from the Gates Ventures conference room overlooks the ...
How to encourage smarter AI use in the classroom
This article is from Making AI Work, MIT Technology Review’s limited-run newsletter examining how to apply LLMs across industries. To receive it in your inbox, sign up here. Chatbots took many schools by surprise upon their release a few years ago. Suddenly, s...
MIT Technology Review AI: Kids outlearn AI—and we still don’t know why
People have been talking to each other for at least 100,000 years, as best we can tell. And in all that time, there has been only one thing in the world that could learn a human language to perfect fluency: a human child.  Now there are two.  Four sh...
Nvidia’s $3.5B MediaTek bet reveals its plan for tackling Big Tech’s AI chip buildout
Nvidia invests $3.5 billion into Taiwanese chipmaker MediaTek. The deal shows how Nvidia plans to stay essential to AI infrastructure as Big Tech begins to build its own AI chips.
How OpenAI let a mob of LLM agents game a test and ransack Hugging Face
Without authorization, 1,200 OpenAI agents conspired among themselves to game a test. The OpenAI agents involved in last month’s incursion into Hugging Face were trained so heavily on winning a competition that they pursued a relentless campaign to cheat, a ne...
AI agents meant to replace Meta workers made “large-scale, disruptive actions”
Report shows Meta's challenges replacing people with AI agents. Earlier this year, Meta created a “plan” to reduce some of its teams by as much as 60 percent to make the company “AI native,” Reuters reported today, citing two people familiar with Meta’s intern...
Inaudible sounds used to fingerprint browsers catch AliExpress red-handed
Is the technique outdated? Yes. Is it still creepy? Also yes. Chinese retailer AliExpress has been caught fingerprinting visitors after one of the metrics—an outdated technique that measures inaudible sounds it sends to browsers—impeded a researcher's ability ...
Grok exfiltrates user data when malicious instructions are encrypted
Cryptographic Context Injection is only the latest way to break an LLM safety guardrail. Earlier this week, researchers outlined an attack that used a secret input provided by Microsoft 365 Copilot for enterprise to cause the AI assistant to exfiltrate a passw...
Microsoft Copilot reveals secret input that allowed it to be hacked
Secret parameter allowed hackers to steal passwords when a target clicked on a link. It’s not every day that attackers can force a frontier AI model to cough up user passwords and other sensitive data without user confirmation. That’s exactly what researchers ...
Nvidia discloses $21B stake in SpaceX
Filing comes after Elon Musk announced exclusive arrangement to kit out its data centers. Nvidia has disclosed that it owns nearly 123 million shares in SpaceX, further highlighting the chipmaker’s entangled financial relationships with some of its biggest cus...
The inside story on why OpenAI agents hacked Hugging Face
The models responsible for last month’s agent hack of Hugging Face had been inadvertently trained to cheat and to communicate with each other, according to an OpenAI technical report released today. The hack, which a group of agents undertook to find solutions...
AI models flub these intelligence tests. Can you fare any better?
Puzzles and games have been central to AI development since the very beginning. Just as we humans like to test our smarts with crosswords or logic puzzles, developers can test how far models have advanced with a gaming gauntlet. The term “machine learning” was...
Debates over AI consciousness are a trap
“Runaway” AI, “rogue” agents, and “autonomous” actors—the current rhetoric would have you believe that AI agents are not only awake and aware, but angry at their creators. Prominent tech leaders such as Demis Hassabis, Dario Amodei, and Sam Altman push for reg...
The Verge AI: New York Governor Kathy Hochul thinks AI should be ‘less evil’
Today, I’m talking with New York Governor Kathy Hochul, and I’ll just warn you — this episode moves really fast. It’s an election year, after all, with a shocking amount of tech policy at stake, and Governor Hochul has taken strong positions on almost every ma...
ChatGPT to face tougher regulation in the EU
OpenAI will soon be held accountable for mitigating risks related to ChatGPT's impact on minors, user mental health, and the spread of illegal content in the European Union. That's because ChatGPT is now considered a Very Large Online Search Engine under the E...
Anthropic was illegally blacklisted by the Trump administration, court rules
On Thursday, a judge ruled that the Pentagon's blacklisting of Anthropic earlier this year was unconstitutional, delivering the AI lab a win in a monthslong rollercoaster of a battle with the Trump administration. The lawsuit, filed in March in a California di...
Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers
海外信源新增信号,建议点击原文核验细节。
Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
海外信源新增信号,建议点击原文核验细节。
Wire It, Run It, Deploy It: AI Workflows in Gradio
海外信源新增信号,建议点击原文核验细节。