Because rationalists distrust mainstream normal science and skepticism, they are far more willing to dabble in mysticism and the supernatural than normal skeptics.

I am also questioning OpenAI’s strategy of strip-mining math for PR, since it seems like advances in math do not actually register that well in the public.

They want to sell the claim that they’ve “achieved AGI”. For the general public they can sell on the vibe of “the computer talks back to you”, but for more sophisticated audiences, they need something firmer. Everyone with even a fraction of the ability to think for themselves has stopped trusting benchmarks, so OpenAI and Anthropic has been looking for things associated with intelligence they can claim success on. Math proofs have the property of being somewhat automatically checkable, having lots of ideas scattered throughout the literature to let their LLMs sift through, and being associated with intelligence.

The /r/singularity and orange site responses have been gross, lots of commenters willing to totally overlook the multiple different ways OpenAI has been utter scum in this debacle in order to celebrate another LLM “success”.

I don’t think it’s a very novel insight

The slatestarcodex branch of lesswrongers have made “high-decoupling” and “steelmanning” and being “gray tribe” their entire identity so its extra funny (compared to normal cults in-group/out-group distinctions) seeing how hypocritical they are at it.

They’ve been talking about the recurrent layers… but my suspicion is that this “innovation” is to obfuscate “thinking” tokens/content instead of actually improving the model in any sort of useful way.

Benchmaxxing for one set of benchmarks can actually degrade performance on other benchmarks. They’ve plateaued for a while, all they can do is scale inference compute up and down (at logarithmically poor rates of exchange) and trade performance in one area for performance in others

A “no attacks on people’s characters” policy seems like a terrible idea

Rationalists: “Its not our fault that we got deceived and manipulated by SBF, lots of people were fooled by him! pay no attention to the fact we repeated the pattern of falling for it with OpenAI and Anthropic also”

Also Rationalists: “You can’t jsut call people in or adjacent to our in-group bad people!”

Also Eliezer: "There’s a standard Internet phenomenon (I generalize) of a Sneer Club of people who enjoy getting together and picking on designated targets. Sneer Clubs (I expect) attract people with high Dark Triad characteristics, which is (I suspect) where Asshole Internet Atheists come from - if you get a club together for the purpose of sneering at religious people, it doesn’t matter that God doesn’t actually exist, the club attracts psychologically f’d-up people. Bullies, in a word, people who are powerfully reinforced by getting in what feels like good hits on Designated Targets, in the company of others doing the same and congratulating each other on it. "

My first thought was maybe one of them isn’t focused enough on doomerism and is too concerned with practical shit, which in lesswrong and AI doomer spaces is a nono (practical concerns about GenAI and data centers are okay as a way to draw in normies, but they want to make sure you speak the shibboleths that indicate your real concern is Skynet). Thus whichever one is the less doomer one would get a smidgen of credit in my eyes.

I was disappointed to learn the real reason…

We endorse pointing out harmful actions. But we condemn attacks on people’s character.

…although it is probably a good thing that someone is willing to call Sam Altman a slimeball and Anthropic lying hypocrites? So I’ll give PauseAI-US a tiny bit of credit for that.

…or maybe not

When I read Holly’s Twitter feed, I get why someone would have a gut reaction like “this is an unhinged person who’s stooping to the level of personal attacks on people who seem nice and are just trying to do their best”.

Whatever credit this PauseAI-US person gets for willingness to call people out, they lose for doing so through twitter.

I don’t exactly why, but based off the dates of various classic lesswrong posts I know it has existed since before 2010 in rationalist culture.

As to inexact reasons why… some key big name starting rationalist (i.e. Yudkowsky for sure, but probably a few others) were into polyamory. (Probably for the obvious reasons of getting to sleep with more people.)

Well, OpenAI and Anthropic have repeatedly degraded their service through unacknowledged quantization and/or dynamically swapping models on the user. /r/singularity has gotten sick of it to the point it is a meme. It is because they can’t actually afford to offer quality SOTA models at the price point people will actually pay for them.

The percentage isn’t total progress to some key point of AI 2027, it is progress relative to their timeline. A constant 1.0 would be staying on track with their predictions, numbers less than that would be falling behind. So they are admitting the real numbers are falling behind their predictions more and more (while still not acknowledging their entire timelines was bs in the first place).

Ed Zitron has also explained his suspicion that lots of GPUs are sitting around in warehouses waiting to be installed, in some cases sold (to juice NVIDIA’s revenue) but not even shipped yet.

And I’m really skeptical speedup from AIs claimed by the LLM companies is in anyway related to reality.

That wouldn’t make sense, because 1.07 would mean progress is negative. I think they meant it as a multiplier? So 1.07 is almost exactly what the predicted, .17 is only 17% of what they expected.

Anyway, it doesn’t really matter, because so much of the input numbers to these calcs are garbage.

That is even worse than using the ARRs, which Ed Zitron has extensively explained are wildly distorted numbers done by shuffling sales and costs around to get really good 30-day to 1-month stretches.

Bioman has already pointed out the “Economic Value” numbers they are using for these tables are probably bullshit (based on self reported run-rate extrapolations that are deliberate distortions at best, based on VC valuation at worst). To add to this… the compute values are also probably bullshit. They are likely based on data center announcements and not confirmed totally complete data centers (Ed Zitron has ripped into how much bs there is in data center announcements). “Coding Time Horizon” is probably METR, which, while some of the best numbers for estimating actual AI improvement for practical purposes, are still really bad in several key ways. (They don’t have enough human task performers for the longer duration tasks even if everything else was right, because they aren’t, and there are several ways systematic bias could have leaked in and compelted distorted the constructed measure of task duration.)

“AI Software R&D Uplift” is the single most important category to their scenario of recursive self improvement… and they have it at a small fraction of what they estimated.

I’ve had the same thought about Factory Porn, it’s is basically what Image slop generators are, right down to the addictive nature, even when it isn’t porn being generated.

Lol, I assume he was just throwing shit at the wall and seeing what would stick?

Skimming the linked nature article… is my understanding correct that you can basically disable the watermarking by turning temperature down to 0?

For example, if the LLM distribution is very low entropy, meaning it almost always returns the exact same response to the given prompt, then Tournament sampling cannot choose tokens that score more highly under the g functions.

Other highlights from the linked paper… to get a true positive rate of 90% with a false positive rate of 1% you need 400 tokens (which should be a few paragraphs worth of text)? (If I’m reading figure 3 right?) That actually isn’t that much, relative to the lengths of essays people write for high school and college classes. …well actually… 1% false positive doesn’t sound too bad, but if you have thousands of freshmen students all taking classes involving writing essays and checking for watermarks becomes the norm, that is dozens and dozens of false positive, which means lots of false accusations, and as we’ve seen from how teachers and institutions have tried utilizing the existing “AI detection” tools that are much much less reliable… I’m getting angry just thinking about it.

Edit: on turning temperature down, it should be noted Anthropic and OpenAI have been increasingly denying the end user internals of their models, such as summarizing or even outright hiding the thinking traces, and not allowing them access to temperature settings either.

This view on the community also explains his recklessness not only with aggressively monetizing HPMOR, but also his willingness to risk drawing JK Rowlings attention to his fanfiction (towards the end of HPMOR, one of his author’s notes was looking for someone to reach out to her).

I would try to get in touch with J. K. Rowling to see if HPMOR could be published in book form, maybe as HJPEV and the Methods of Rationality, with all profits accruing to a UK charity. I’m not getting my hopes up, but I do have a rule telling me to try rather than automatically giving up and assuming something can’t be done. If any reader thinks they can put me in touch with J. K. Rowling, or for that matter Daniel Radcliffe, regarding this matter, I do hereby ask them to contact me at [email protected].

Rowling (and Warner Brothers) have been litigious with their IP: https://en.wikipedia.org/wiki/Legal_disputes_over_the_Harry_Potter_series#Other_accusations_of_infringement . So this was a reckless move on EY’s part, possibly indicating that he doesn’t appreciate the delicate place fanfic (and its communities) are in.

Edit: thinking about the way Rowling has shown her true face over the past decade… “with all profits accruing to a UK charity” makes me wonder if in whatever fantasy universe Eliezer did talk Rowling into allowing published fanfic to be monetized for “charities”, that money would all end up going to TERF “charities”.

One of the plot threads was about the invention of AI, in the Data-from-Star-Trek kind of sense.

Going on a tangent from this… Its funny how much AI-related sci-fi is feeling increasing quaint (and often foolishly optimistic).

You know what depiction of AI has held up, and maybe makes even more sense in light of modern AI? Star Wars’ take on Droid. Prior to LLMs it seems obvious that droids were sentient and deserving of rights. Post LLMs… it seems pretty plausible you could get something like C3PO that blabbers on but lacks any meaningful sentience or sapience. Also, it seemed obviously idiotic the way the Trade Federation had humanoid droids acting as pilots and gunners instead of building proper autopilots and targeting system. But after seeing LLMs get shoved everywhere, I totally could imagine a megacorp shoving droids built from standardized pretrained modules into all kinds of applications they aren’t fit for.