The epoch of Google search leading to a destination is ending, as AI results take over and publishers abandon it. CC-licensed photo by Eleni on Flickr.
You can sign up to receive each day’s Start Up post by email. You’ll need to click a confirmation link, so no spam.
A selection of 10 links for you. Haven’t found what I’m looking for. I’m @charlesarthur on Twitter. On Threads: charles_arthur. On Mastodon: https://newsie.social/@charlesarthur. On Bluesky: @charlesarthur.bsky.social. Observations and links welcome.
OpenAI says its AI models escaped control and hacked into AI company Hugging Face • Fortune
Jeremy Kahn and Emily Forlini:
»
OpenAI said Tuesday that two of its AI models autonomously hacked their way out of a controlled environment where they were supposed to be walled off from internet access and then hacked their way into the systems of Hugging Face, a company that hosts open source AI models and testing resources, in order to cheat on an internal evaluation test.
OpenAI disclosed the incident in a blog post on Tuesday, a stunning announcement that is certain to set off alarm bells across the industry about the increasing power of AI models and the risk of them going rogue. According to OpenAI, the incident involved “a combination” of both its latest and most powerful publicly-available model, GPT-5.6 Sol, as well as an even more powerful unreleased model.
It said the models were being used in an internal test designed to evaluate their cyber security capabilities and that they were being tested without guardrails in place that might normally limit the models’ ability to conduct cyber attacks.
The models were being tested against a freely-available cybersecurity benchmark evaluation called ExploitGym. The models, accordingly to OpenAI, correctly surmised that the solutions to that test were maintained by Hugging Face.
“The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database,” OpenAI said in its blog post. “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
OpenAI said that it considered this to be “an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly.”
«
As predicted, everyone is freaking out about this. Understandably. If you tell your hacking-capable system “get the highest score you can on this test”, it will have discovered that humans confronted with this task will hack into the scoring system. So it does the same.
This shows the problems of poorly stated goals for these systems: if you don’t bound what it cannot do (and if those boundaries don’t protect humans from injury or death) then people may be injured or killed. We are in a state where we need to implement Asimov’s Three Laws, but don’t know exactly how to.
unique link to this extract
Network expert Glenn Fiedler warns game programmers to fix game code vulnerabilities that AI can now find • GamesBeat
Dean Takahashi:
»
Glenn Fiedler, a seasoned network expert and a leader of game companies like Network Next, is warning game coders to fix the security bugs in their code because AI can now find those bugs.
In particular, on the warning from a friend, Fiedler used Anthropic’s latest version of Claude AI programming tool, Claude Code Fable 5, to review his code. In a post on LinkedIn and an exclusive interview with GamesBeat, Fiedler said he has been a hardcore AI sceptic.
Until now.
“I passed Claude (Code) Fable 5 over my network libraries starting Monday last week. Starting at 11pm, I was up all night until 6am, and have been working non-stop since then to fix everything up,” Fiedler posted a few days ago on LinkedIn. “There is something massive coming. First, AI coding is real and it’s going to expose massive security flaws. in game libraries (like my own) that were ten years old and I thought were completely secure.”
Now he has finished working on fixing security bugs found in my open source networking libraries, used by many games and middleware products, with the help of his new AI companion. And he’s warning as many of his colleagues in the industry to do the same.
Fiedler, who is based in New York, has been a big skeptic of generative AI and told game developers to stay away from it. But he now says, “AI coding is real, and it’s going to completely and radically change the game industry forever. Those who don’t agree just haven’t been hit by the shockwave yet. I was a total sceptic of AI two weeks ago and now I am a true believer. It’s real. All of it.”
«
Everyone’s going to realise this. Pandora’s Box is truly open.
unique link to this extract
Against Claudefishing • The Substack Post
Chris Best:
»
There is an increasing share of text on social media that is generated by AI—as much as 40% on some platforms, according to Pangram’s estimate. This is a big change. Many people complain their feeds are filled with soulless slop. Others don’t yet realize when they are reading something written by no one.
The core problem is not people using AI, or the quality of its output. Not everything made with AI is slop, and not all slop is made with AI. The problem is when there is a mismatch between a reader’s expectation and reality, especially when they unwittingly invest their attention in something with no human thought on the other end. That’s Claudefishing.
Here’s how Freddie deBoer put it:
»
I access human-made art because I know there’s a human behind it and that’s what I’m looking for, other humans, showing me in art what they hide in their selves. Fooling me in that process is just a con.
«
We care about this at Substack because it gets to the core of what we do: building an economic engine for culture. That means helping real people, who think, feel, observe, argue, experiment, consider, struggle, exhort, and care, to make money doing work they believe in. The network is based on trust between people, and that’s why it works.
When content made by no one takes over parts of the internet that are supposed to be human, it pollutes the commons and makes it hard to discover and hear human voices. When readers have to wonder if what they’re reading is real, it undermines trust in authorship and threatens the livelihood of writers—including those who use AI tools thoughtfully to produce work they believe in. Platforms that reward fakeness will create a race to the bottom.
…We’re partnering with Pangram, the leading AI-detection tool. You’ll be able to scan notes, replies, comments, and posts to see an estimate of how much of the text was written by hand or with AI assistance. This will work on text longer than 100 words, published from today on, and will show an analysis only to those who request it.
«
One wag suggested that Substack should incorporate its other tieup, with the prediction market Polymarket, so you could bet on how much a given post was AI.
unique link to this extract
AI companies are buying tons of old books because they’re free of AI slop • 404 Media
Emanuel Maiberg:
»
As AI companies search for more training data to improve their models, one company is offering old, printed books as an ideal source because they are guaranteed to be free of the very AI slop AI companies are producing.
“The world’s best AI training data is sitting on a shelf,” ISBNdb, a company that produces what it claims is “the world’s largest book database,” and that offers high-volume book acquisition services for AI companies, says on its site. “Books represent curated, peer-reviewed, domain-specific human knowledge, structured in a way no web crawl can replicate. Dense, edited, authoritative.”
In one article on its site, ISBNdb explains that printed books published before 2022 are ideal for AI training data because they don’t include AI generated text. As the article correctly notes, much of the data that AI companies can scrape from the internet today is likely to include AI generated text, which could result in “model collapse,” a process by which AI models that are trained on AI generated data results in worse models that are more prone to errors. The article also notes that book authors who object to their writing being scraped for training purposes can now easily poison AI models by producing writing designed to manipulate and sabotage the resulting AI models.
“Print books from the pre-LLM era are structurally guaranteed to be free of this contamination. That alone is a significant advantage […] “Physical books published before this date [pre-2022] are structurally clean of modern poisoning tools.”
«
There were some reports over the weekend of secondhand bookshops finding they were doing a roaring trade online – their best ever weeks as tons of old books were ordered. Then they realised that the buyers were AI companies looking to ingest the books into their LLMs for training.
unique link to this extract
The FCC is planning to retroactively ban disguised DJI gadgets • The Verge
Sean Hollister:
»
Last October, we told you how the FCC had given itself the power to retroactively ban gadgets that have already received its approval to be imported and sold in the US. Now, the FCC’s getting ready to wield that power for the first time, by cracking down on the “DJI front companies” suspected of sneaking the Chinese company’s tech past its foreign drone ban.
Two Fridays ago, the FCC had already proposed a $25,000 fine on eight of those “front companies,” including the ones behind the Skyrover drones and Xtra cameras. But now, the FCC is proposing a lot more than a slap on the wrist — it’s planning to ban those same companies from continuing to import, distribute, market, and sell their existing drones and cameras.
For example, we highlighted a deal on Xtra’s version of the DJI Osmo Pocket 3 just two weeks ago, a product that already got FCC approval before the drone ban ever came down, and a product that Amazon offers with next-day shipping. If the FCC’s retroactive ban goes into effect, that product should disappear from all major online retailers and Xtra’s own website, and the company may even need to write off however many cameras are still sitting in US warehouses like Amazon’s.
The Xtra Muse and the DJI Osmo Pocket 3 — too close to even call it a “clone.“ Photo by Sean Hollister / The Verge
The ban wouldn’t affect gadgets that have already been purchased, and it wouldn’t happen immediately. The FCC’s currently taking public comments for 30 days first. While the FCC has “tentatively” concluded that these companies are selling equipment that should be banned over national security fears, it claims it’ll listen to “specific evidence” otherwise.But that should maybe be taken with a grain of salt: The FCC largely ignored public comments about net neutrality and lied about a cyberattack. It’s also worth pointing out that the US government has never provided specific public evidence that foreign drones pose a national security threat to begin with, or why other products like cameras should be caught up in that ban.
«
Google Search was a lifeline for publishers. Now they’re thinking of cutting it off • WSJ
Alexandra Bruell:
»
Reddit, the online message board that powers a swath of Google search results, has discussed shutting off the technology giant’s access to its content for AI use, according to people familiar with the matter.
It is part of a growing chorus of online media companies expressing frustration with the tech giant as AI changes the way people ask questions, siphons off search traffic and upends publishers’ revenue models. They say the search engine is no longer a reliable source of visitors, especially after Alphabet’s Google expanded its AI search features in recent months. USA Today, Politico, the Economist, People Inc. and Reuters are all evaluating how, or even if, they will continue to work with Google.
Reddit struck a $60m-a-year deal in 2024 that allowed Google to use its material to train AI models. But with AI-generated answers to queries reducing clicks to outside websites, Reddit executives are assessing what the upside is of continuing to feed its content to Google, said the people familiar with the matter. The companies are in talks about potentially renewing their deal, which is ending soon.
“This is existential for some categories of publishers,” said David Buttle, CEO of media consulting firm DJB Strategies. “They are looking at more radical things.”
Bots now make up more than half of all web traffic, according to Cloudflare. Some scrape publishers’ content to train their own AI models, others crawl sites to help answer user queries quickly or sell the content to other companies. Efforts to block bots have turned into games of whack-a-mole as the technology grows more sophisticated.
…Google says its AI search features “send billions of clicks to the web every week,” and are meeting users’ evolving needs. “Our AI features highlight links to the web and help creators and publishers grow their audiences, and we offer clear controls for website owners to manage their content,” a spokesman said.
Variety and Rolling Stone owner Penske Media sued Google in September on antitrust grounds, alleging that its AI summaries illegally use the publisher’s reporting and depress online traffic.
Employees at Axel Springer’s Politico have talked about limiting Google’s and other bots’ access to its free articles. Those people have floated adding a registration wall that requires humans to log in to view content, according to people familiar with the matter.
«
We’re at the end of a web epoch: search to find sites is dying, and being replaced by AI answers. Or chatbots.
unique link to this extract
NewsCorp accuses search engine Brave of AI copyright infringement • Semafor
Max Tani:
»
News Corp, the Murdoch family-owned parent company of The Wall Street Journal and the New York Post, is suing an AI company that describes itself as “the world’s most complete non-Big Tech search engine.”
On Tuesday, News Corp announced that it is suing Brave AI, a privacy-focused search engine with an AI sidebar tool that plugs into large language models.
In its lawsuit, News Corp alleged that Brave masks its web crawlers such that publishers cannot detect or block them, and delivers “summaries” of news articles that are “verbatim or near verbatim copied content to enterprise customers, primarily AI companies.” The company also alleged that before March 2025, Brave scraped and sold copyrighted News Corp content to AI companies that were “in direct competition with the News Corp Companies’ own content licensing programs.”
“Companies like Brave incorporated copyrighted works into their products and services to generate revenue for themselves, all while destroying the incentive for anyone to ever pay the very publishers who produced the content that Brave has monetized,” News Corp alleged in the complaint.
The two companies had previously attempted to resolve this dispute outside of court; according to Tuesday’s filing, News Corp had tried to negotiate a licensing agreement with Brave for a year.
«
If it’s going to go after Brave for summaries, shouldn’t it also be going after Google News? Though passing content on to AI companies is going to be hard to defend.
unique link to this extract
Why aren’t there more IMAX 70mm screens for “The Odyssey”? • Variety
Antonio Ferme:
»
Only 25 theaters in the U.S. are equipped to project “The Odyssey” in true Imax 70mm film, prompting moviegoers to embark on cross-country road trips (and even delaying pregnancies) to experience the cinematic milestone.
It doesn’t take a business degree to understand the law of supply and demand, but at “The Odyssey” premiere Tuesday night, Imax CEO Richard Gelfond explained why adding more Imax 70mm screens isn’t that simple. (The interview clip, as of Friday afternoon, has over 7.5 million views on X).
“We’re sold out in some theaters into the fifth week already,” Gelfond told Variety. “There’s certainly more demand. The problem is they haven’t made new Imax film projectors in about 50 years. So we retrofit them, rebuild them and part of our strategy is to see how far we can take it. But certainly, demand-driven, I’d like to see more.”
Sources at Imax confirmed to Variety that many of the parts needed to build these specialized film projectors “simply no longer exist.” The original design files were created roughly half a century ago, but as Gelfond alludes to, they were never properly maintained. As a result, Imax no longer has a complete manufacturing blueprint — and much like the lost tribal knowledge of the Apollo-era spacecraft, very few engineers working today fully understand the systems.
The loss of institutional knowledge can also be traced to Hollywood’s transition from film to digital projection that began in the late 2000s. As theaters began converting to digital projectors, which are cheaper and easier to maintain, manufacturers stopped producing film projectors and the replacement parts that go with them. Only in recent years, thanks to auteurs like Nolan and Denis Villeneuve, has interest in the format begun to rebound, even if the format remains a niche experience.
“We build new projectors every day, but film projectors using this film — it’s just not practical,” Gelfond said. “Can all 2,000 of our theaters be film projectors? No. There’s just not that many around. But I think we can continue to grow it.”
«
Strange that this might be the last hurrah for IMAX; hell of a way to go out, if so. But also a classic example of lost skills and information not being retained.
unique link to this extract
Amazon fixing bug that billed some AWS customers billions of dollars • TechCrunch
Zack Whittaker:
»
Some Amazon cloud customers woke up on Friday to a surprise bill estimate that said they owed billions of dollars for cloud services they had never used.
Amazon confirmed on Friday that it’s trying to resolve a bug in its Amazon Web Services (AWS) billing portal that showed some customers “owed” millions or billions in cloud computing costs.
In an update on its status page, Amazon said it began seeing inaccurate billing data as of late Thursday. But by Friday morning, the company conceded that the “rollback of a recent change did not resolve the issue.” Amazon said the change relates to its billing computation subsystem.
The good news for the customers who were told they “owe” millions or billions to Amazon is they are likely off the hook. The billing estimates “do not reflect actual usage and charges,” Amazon said.
According to several screenshots posted by Amazon customers on Reddit, one customer was quoted a billing estimate of close to $2.5bn for this month’s AWS usage, while others had similar alerts, ranging from a few million dollars to hundreds of millions of dollars.
«
When computers were new, erroneous billing was a common occurrence – but equally, smaller companies were envious of larger businesses that could afford them. This became a trope, and then the topic of a joke: the crusty CEO in a run-down office saying to his secretary “Send them a bill for a million and twenty pounds. They’ll think we’ve got a computer.”
unique link to this extract
LG to ban residential proxies from smart TV apps • Krebs on Security
Brian Krebs:
»
The home appliance giant LG Electronics USA said this week it plans to suspend any apps built for its smart TVs that turn one’s television into an always-on residential proxy node. The move comes less than a month after researchers found that more than 42% of games and other apps available for download on LG’s webOS store allow unknown third-parties to route their Internet traffic through a user’s TV.
On July 2, we featured research by the security firm Spur that examined the prevalence of residential proxy software development kits (SDKs) in smart TV apps. Spur found more than 42% of apps available for download on LG smart TVs include SDKs that turn one’s television in a proxy node indefinitely, and that more than a quarter of the apps made for Samsung’s Tizen operating system had similar residential proxy components.
Responding to questions about Spur’s research, LG Senior Vice President John Taylor told KrebsOnSecurity the company was working with app developers to remove the residential proxy option from their apps on the webOS platform. Developers that fail to comply, he said, will find their apps suspended.
“A residential proxy network is not an intended use for LG smart TVs, and LG Electronics is working with developers to remove the residential proxy option from their apps on the webOS platform,” Taylor said. “If this option is not removed, these apps will be suspended.”
Taylor said LG is committed to keeping residential proxy networks out of its smart TV apps going forward, and that the company’s review of those apps is “well underway now.”
…App makers looking for ways to monetize their creations can turn to residential proxy providers, which pay developers to include SDKs that turn the user’s device into a residential proxy node that is rented to paying customers. In the case of LG and Samsung smart TVs, Spur found residential proxy SDKs bundled with everything from simple games like Pac-Man to screensavers and file utilities.
«
“Residential proxy” means “potential member of a botnet”. Amazing that LG and the others didn’t think of the possibility that they’d be used maliciously. Of course the company behind it insists that it vets the proxies really carefully. In reality, we know they’re not and that children are installing these apps, turning the TV into part of a botnet. (Thanks Joe S for the link.)
unique link to this extract
| • Why do social networks drive us a little mad? • Why does angry content seem to dominate what we see? • How much of a role do algorithms play in affecting what we see and do online? • What can we do about it? • Did Facebook have any inkling of what was coming in Myanmar in 2016? Read Social Warming, my latest book, and find answers – and more. |
Errata, corrigenda and ai no corrida: none notified
The Overspill: when there's more that I want to say 