Working with an LLM can be like getting caught in one of Zeno’s paradoxes, where finishing is impossible. CC-licensed photo by Conrad Bakker on Flickr.
You can sign up to receive each day’s Start Up post by email. You’ll need to click a confirmation link, so no spam.
A selection of 9 links for you. Slowly coding. I’m @charlesarthur on Twitter. On Threads: charles_arthur. On Mastodon: https://newsie.social/@charlesarthur. On Bluesky: @charlesarthur.bsky.social. Observations and links welcome.
COLDCARD wallet RNG flaw likely linked to $88m Bitcoin theft • Bleeping Computer
Lawrence Abrams:
»
Researchers suspect that a vulnerability in COLDCARD hardware wallet firmware was exploited to steal an estimated $88.6m in Bitcoin from thousands of wallets whose seeds were generated using a flawed random number generator.
Digital asset research firm Galaxy Research says it identified an initial wave of transactions that it believes was likely linked to the vulnerability, draining approximately 1,083 BTC, worth $70.2m, from 1,196 addresses on July 30.
The 41-minute attack occurred approximately 30 hours before Coinkite publicly disclosed the flaw.
Every transaction used an identical hardcoded fee rate of 30 satoshis per virtual byte and left no change output, making Galaxy believe the attackers used an automated tool.
“Signature: every sweep paid an identical hardcoded 30.0 sat/vB — a 30-75x overpay vs the 0.4-1.0 sat/vB median that week — and left no change output, explained Galaxy.
“That looks like an automated tool spending keys it already held, not owners moving funds.”
On August 1, Galaxy Research identified a second and third wave, raising the estimated total to 1,367 Bitcoin, worth approximately $88.6m, stolen from 4,585 addresses. The stolen Bitcoin remained in the attacker-controlled addresses at the time of its report.
Chainalysis found that the attacker prioritized high-value wallets, stealing approximately $30m during the first ten minutes and taking $1.8m from one victim.
The company said this suggested the attacker had identified and studied the affected wallets before beginning the thefts.
«
The money is all sitting in one address, which will be closely watched. But there are ways to obscure transactions, and if this is a state-sponsored hacking then there will be friendly “mixers” who in effect launder it. This explains how the coding mistake was made (an impatient programmer trying to clear a compiler error).
Once more the folks who celebrate money that’s free of the evil financial system and government control are calling for the financial system and police to help them out. That seems optimistic.
unique link to this extract
Inside an AI TikTok shop slop factory that shills supplements recalled by the FDA • 404 Media
Jason Koebler:
»
In February, a content creator from New Zealand named Harry Chang posted a YouTube video called “This AI TikTok Shop Video Made Me $67,420 (Here’s How).” In the video, Chang and another YouTuber, Jimmy Farley, describe how Chang created a viral marketing video for a supplement company called Rosabella called the “Nigerian SECRET to CLEAN LIVER!!”
Chang explains that he copy-pasted the script from a video posted by an account called “liverboosthub11” and tweaked it to suggest the supplement he was marketing is “something that’s been used in Asia or Africa for a long time that a lot of people don’t know about in America.”. In Google’s VEO 3, he created an AI-generated Black woman wearing a surgical mask in the foreground of the video, pointing up at another AI-generated Black woman (created in a tool called HeyGen) wearing a pink dress and standing on a stage. He directed the woman in the foreground to have a “strong African American accent,” and to say, “Why is nobody talking about what this hoe said?! If you’ve got issues with that belly, must watch!”
“Builds curiosity, builds intrigue,” Chang says about his creation. “People want to know, ‘damn, what did she say? What did that ho say?’” To make the AI-generated woman on stage read the script he took from liverboosthub11, he pulls up the popular AI voice generator ElevenLabs. He scrolls through a list of voices that he had created, including “African American Woman Organic,” “Black Man 1,” “ORGANIC White Southern Woman,” and “Sad Black Woman in Car.” He settles on a voice called “Latisha 1.” He syncs the voice with the AI-generated videos he created in the video editing software CapCut. This AI video goes on to get 1.3 million views on TikTok and apparently earned him tens of thousands of dollars in affiliate sales.
In another video, Chang explains how he has made tens of thousands of dollars using AI influencers. “I’ve even had my clients buy me Rolexes for selling so much of their products,” he says.
The video’s title is “How I print $51,000/month profit with AI influencers (feels illegal).”
A new lawsuit argues that the strategy is, indeed, illegal. (After 404 Media asked for comment for this story, several of the YouTube videos mentioned in this article were deleted).
«
Fantastic payoff line there, though taken as a whole, the situation is – as with so many of these things! – dispiriting.
unique link to this extract
The real reason that Substack is collapsing • Magnetic North
Scott Carney:
»
A few years ago Substack transformed away from being a niche newsletter-delivery system and into the main refuge for tens of thousands of journalists who were fleeing (or been kicked out) of the mainstream media. The value proposition was simple: writers could come here to build an audience that would actually pay directly for their writing. The platform recommended writers put paywalls on posts that would encourage readers to financially support their favorites. The tradeoff from other social media platforms was that the paywalls would inherently limit the reach of their best work. Nonetheless, with a thousand paid subscribers you could live the middle class dream of earning close to $100,000 a year (well, more like $80,000 after all the fees).
For a while it seemed like a reasonably good trade. No one wanted journalism to die, and the new influx of writers was doing pretty good work on the platform—albeit with lower budgets than traditional newsrooms.
But the model raised an obvious question: how many individual newsletters could the platform rationally expect readers to pay $8/month to access?
Early adopters on the platform did the best. Writers like Heather Cox Richardson and Matthew Yglesias arrived early and earned millions. But as more writers showed up the subscription revenue pie didn’t grow commensurately. Even with Substack’s discoverability mechanism (something that is missing on places like Ghost and Beehiv) it was getting more difficult to establish a real career here.
So Substack tried to solve the problem by adopting a strategy that every other tech company had already worn thin. They pivoted to adding a twitter-like function called “Notes” (which created a distinction between ““subscribers” and “followers”). Soon they incorporated video and live-streaming and encouraged posters to clip shorts into their feeds with an AI-generated clipping program.
In other words: Substack’s growth strategy was to offer almost the exact same things as every other social media network. They pushed the idea that the subscription revenues would keep the best creators on this platform and the new engagement opportunities would grow truly massive audiences. The stated plan was to make Substack the go-to-place for everything internet in a winner-take-all contest.
At first the strategy even sort of worked. A lot of new users DID join the platform and engaged with all this new free content—increasing the overall number of eyeballs that were here. These new “engaged users” helped skyrocket the VC-backed valuation , but, notably, didn’t do much to help the flood of new creators who came here to actually earn more money.
In fact, they started earning a lot less.
«
Insightful piece. The question this made me ask was: when will Substack’s VC investors start demanding their returns? If we assume Substack is bumping along OK on its cut from subscriptions, is that enough to make it profitable? Can it grow? Is it going to try to go public (which would be disastrous, surely; wrong incentives everywhere)?
unique link to this extract
Why is the world so loud? • The Atlantic
Bianca Bosker:
»
A resident of Chandler, Arizona called Karthic] Thallikar went hunting for the source of the sound. At first he canvassed the neighbourhood by foot, setting out around 10 or 11 o’clock at night, once the thrum of traffic had quieted down. When these “noise patrols,” as he called them, yielded no answers, he expanded his perimeter—by bike, then by car. He’d pull over every few blocks to listen for the whine. The hum was everywhere: outside Building E of the Tri-City Baptist Church and the apartments in San Palacio; near the Extra Space Storage and the no perfect people allowed sign at Hope Covenant Church; ricocheting around the homes in Canopy Lane, Clemente Ranch, Stonefield, the Reserve at Stonefield. He’d go out multiple nights a week, for 10 minutes to an hour, taking notes on where the noise was loudest. The patrols dragged on—one week, two weeks, eight weeks—which led to spats with his wife, who wanted to know why he kept leaving the house so late at night.
Finally, as winter warmed into spring, Thallikar thought he’d identified the source of the whine: a gray, nearly windowless building about half a mile from his house. The two-story structure, which had the charm of a prison and the architectural panache of a shoebox, was clad in concrete and surrounded by chain-link and black-metal fences, plus a cinder-block wall. It belonged to a company called CyrusOne.
There was no thrill in this discovery, just simmering fear that the noise might get worse. Thallikar visited the city-planning clerk, multiple times. She said she couldn’t help and referred him to CyrusOne’s construction manager. Kept awake by the noise at 11 o’clock one Saturday night, Thallikar phoned the man, who protested that he was trying to sleep. “I’m trying to sleep too, dude!” Thallikar told him. When they spoke again the next day, the call ended abruptly, and without resolution.
According to CyrusOne’s website, the company’s Chandler campus offers Fortune 500 companies robust infrastructure for mission-critical applications. In other words, it’s a data centre—a columbarium for thousands of servers that store data for access and processing from virtually anywhere in the world. When you check your bank balance or research a used car or book a hotel room, chances are decent that the information comes to you via one of the more than 40 CyrusOne data centers spread around the globe. CyrusOne houses servers belonging to nearly 1,000 companies, including Microsoft, Country Financial, Brink’s, Carfax, and nearly half of the Fortune 20.
Thallikar, wanting to confront the noise personally, made a surprise visit to CyrusOne.
«
That was back in 2014-2015. As you might imagine, Thallikar isn’t any happier now. (Gift link.)
unique link to this extract
The Zeno’s Paradox of AI • Anuroop Bisaria
Anuroop Bisaria:
»
Ask a coding agent for a one-line fix. You get the fix, and then a question: want me to also add tests? Say yes and the tests arrive with an offer to wire up CI. A few yeses later you’re refactoring the deploy pipeline behind a typo fix, and there goes your afternoon. I use these tools every day and I have never once reached the bottom. There is always one more thing.
Zeno’s paradox has Achilles chasing a tortoise: by the time he reaches where it was, it has moved on, forever. Therefore Achilles can never catch the tortoise. It describes working with a language model annoyingly well. The task goes 90% done, then 95, then 97.5. The limit is right there. You’re never getting to it.
That’s the thing I want to name: under the incentives these models are trained and sold under, an AI can finish your task but it cannot close it. Something is left open every time. On purpose, mostly.
…Some of it is inherited. The training corpus is full of call-centre English (“is there anything else I can help you with today?”) and consulting decks, and every consulting deck in history ends on a Next Steps slide. The models learned the register from us.
Preference training makes it worse. Human raters reliably favour longer answers and offers of further help, so the reward model does too. The penalties are asymmetric on top of that: say “done” when it isn’t and you get burned; hedge when you didn’t need to and nobody notices. A model that never claims completion is making a rational bet.
There’s a mechanical detail I find clarifying. Ending a turn is a literal token the model has to choose to emit. Nothing in the reward wants the token that ends the conversation.
The deeper cause: “done” is a property of a whole task, and nothing in the training loop ever sees a whole task.
«
Apple struggles to keep pace with AI bug hunters • Financial Times
Tom Wilson and Michael Acton:
»
Apple has restricted the number of potentially dangerous software bugs researchers can submit to its internal security team, as it faces a deluge of reports from people using AI models to identify alleged risks.
The Cupertino-based tech giant told the FT it had moved in June to limit the high volume of requests it was receiving, with its review system coming under pressure from “AI slop” reports that can hallucinate security risks in its software.
Apple is grappling with an industry-wide phenomenon that has resulted in generative AI software tools transforming the cyber security arms race, with an increase in the detection of real security flaws and a wave of poor-quality submissions from amateur bug hunters using AI, the company said.
The change in Apple’s approach was highlighted by Italian cyber security startup Bynario, which told the FT it had used OpenAI’s ChatGPT to identify more than 50 bugs in the latest version of the MacOS operating system in just three weeks.
Among them was one of the most serious types of vulnerability, a so-called privilege escalation exploit chain, which could allow an attacker to seize full control of an Apple computer by gaining unrestricted access to the system.
However, the startup said it was unable to alert Apple to the vulnerability because the tech giant had limited the number of bug reports it could make.
«
People who have been using Apple’s betas this year have been praising them while also noting that the company seems to be treating this as a “fix it, forget new features” year (though of course there are new features). That’s almost certainly due to greater use of AI coding inside Apple – which will also have access to coding LLMs, just like Bynario, and so might be able to find the same bugs and chains of hacks.
unique link to this extract
A PR firm is using fake publicists with AI-generated headshots to spam journalists with pitches for its clients • Futurism
Maggie Harrison Dupré:
»
Does it ever feel like your inbox is filling up with emails that don’t sound quite human?
You may be right. Last week, I received an email from a publicist identified as “June Barton,” whose address indicated that she worked at a firm called VectorGuideHQ.
The message, which read like any one of the countless PR emails in my inbox, was seeking coverage for a wellness app called RiseGuide. Barton offered to set me up with an account and introduce me to the startup’s founder. When I didn’t respond, Barton followed up.
“Glad to set you up so you can try it, or put you on a call with the founder,” Barton wrote. “Lmk!”
RiseGuide is a real company. But June Barton is not a real PR rep, or even a real person.
For one thing, her headshot — showing a light-eyed, soft-smiling brunette — was clearly AI-generated. Though not visible in the tiny email window, when I opened her profile picture in a new tab, I could see that the image had failed to cut out a watermark disclosing that it was downloaded from the site “this-person-does-not-exist.com,” which traffics in AI-generated faces.
“Generate random human face in 1 click and download it!” reads this-person-does-not-exist.com. “AI generated fake person photos: man, woman or child.”
Even more peculiar was that when I looked up the VectorGuideHQ site in Barton’s email address — june@vectorguidehq.com — it redirected to a United Kingdom-based PR company called Movchan Agency. Barton, I soon learned, was one character among a roster of fake personas used by Movchan to pitch journalists on stories.
When I reached out to the firm’s clients, they said they were unaware of the practice.
«
As a journalist, this would be a personal version of Hell. Or possibly I’d just stop reading email altogether.
unique link to this extract
How to plant a nuclear plant in Iran • Digital Digging
Henk van Ess:
»
Tonight I typed just one sentence into Google Earth and put refugees near the Mexican border. Then I planted a nuclear plant in Iran. Then I put a fatal crash on a street in Amsterdam. Google’s own satellite imagery underneath all three. What on earth is Google doing?
Google spent twenty years building the reference the world checks against.Today it added a button that makes things up.
Image generation went live in Google Earth on the web this afternoon, powered by Nano Banana 2, worldwide, for everyone, with no waitlist and no application. You zoom to any coordinates on the planet, click create image, and type what you want to see. The model takes Google’s satellite, aerial and 3D imagery of that exact spot as its starting point and hands back a photorealistic picture. If it isn’t quite right, you refine it.
I added a hospital and a bomb crater in Gaza in seconds with Google Earth. I can’t believe I just typed this.
Google describes the feature as creating “concepts grounded in the real world,” which is the problem stated as a selling point. Grounded means the invented thing is welded to genuine coordinates and drawn on genuine imagery, often in the same colours and the same light and at the same angle as the picture beside it. A forgery built that way doesn’t have to be convincing on its own. It inherits the credibility of the map it was born on.
The announcement runs to 487 words, written by Bryan Horowitz, Product Manager, Google Earth. Horowitz gives the planet a six-word instruction: type whatever you want to see. That is the whole gate. Elsewhere in the same post he suggests it’s fun to let your imagination run wild, and the piece signs off by inviting you to pick a spot on the map and start bringing your ideas to life.
Somebody is going to bring an idea to life before morning. It will not be a community garden, trust me.
The people who want this are not hard to picture. A government official wants a strike to look bigger than it was. Two factions want the same hospital to look flattened or intact, depending on which of them is holding it this week. A troll wants forty thousand reposts before lunch. This morning all three needed a screenshot, a second browser tab and a little patience. Tonight they need a sentence.
«
Google rather quickly rolled this back but then said it looking to install “better guardrails” rather than what it should be doing, which is scrapping the whole idea. Why wreck any reputation you (Google) have for accuracy by letting people create AI slop? Google said there would be a watermark. Don’t bet your life on people not beating that.
unique link to this extract
AI market correction emerging as major credit risk • Fitch Ratings
»
The scale of the AI investment boom and the accelerated global technology cycle has been a significant driver of US equity market valuations and corporate bond issuance over the past year. The effects on real economic indicators are profound. The 18% yoy rise in IT capital investment directly added 1.4pp [percentage points] to 1Q26 GDP growth. The wealth effect from AI-related investor optimism and equity market gains has also been a meaningful support for US consumer spending growth, which has been broadly slowing.
That said, the medium- and long-term potential of the underlying technology is highly uncertain, as with previous tech cycles. The combination of revenue uncertainty and the extent to which capital markets and economies have become intertwined with AI have created a vulnerability for credit in the event of a re-evaluation of long-run returns potential. Very short-term spikes in market volatility for individual equities and tech-heavy stock indices have already occurred, but a larger, more protracted correction could have wider market, macro and credit effects depending on its scale, duration and contagion.
«
Fitch, the ratings company, didn’t call the 2007/8 crash ahead of time, so it’s aiming to get in front of this one.
unique link to this extract
| • Why do social networks drive us a little mad? • Why does angry content seem to dominate what we see? • How much of a role do algorithms play in affecting what we see and do online? • What can we do about it? • Did Facebook have any inkling of what was coming in Myanmar in 2016? Read Social Warming, my latest book, and find answers – and more. |
Errata, corrigenda and ai no corrida: none notified
The Overspill: when there's more that I want to say 