The Chinese government has stopped direct subsidizing solar panels years ago. I think it was around 2019? This resulted in a lot of companies going under at the time.
It did not stop solar panels getting cheaper and cheaper because of the whole integration and mass production (with healthy free market competition).
The last subsidies like export value-added tax rebates for solar panels and lower rebates for batteries are ending in 2027.
China their main power is, the ability to have everything inhouse. Yea, they subsidize a lot of stuff until it hits critical mass, and then you have often a healthy industry with lots of competition.
China alone has like a few 100 car manufactures because of the subsidies, and over time there will be consolidation / buyouts etc but the end result is a healthy new industry that exports. With again, everything internally being produced.
This is why our subsidies fail. We do one sector, often a few companies at best. This results in few competitors, expensive prices, and often reliance on externals that can bankrupt those companies. And que how we wasted again dozens of billions in propping up a industry with no competitive edge.
People can cry about China but they are actually doing work, despite the mass amount of corruption. That is the big difference with here... Mass corruption got in the way of national security, plop, people go to jail. Industry quickly gets their ** together. Here ... give billions, and the money vanishes, with no real consequences.
"GLM 5.2 is just shy of GPT 5.4"... If your running the full model. As in have 750 (FP8) to 1.5TB(FP16) of memory available.
Do not mix the benchmark results of GLM 5.2 FP16/FP8 with FP4 or FP2.
* FP4 will mean a accuracy loss of about 3%. Not noticeable but more chance for mistakes.
* FP2 ... what is what most people are able to run at home, for a "reasonable" price. Your looking at over 17% loss in accuracy.
At that point, your running at less then claude-sonnet-4.6, as the issues compound with accuracy losses. And reasonable priced is still in the ~ $5000 range (192GB + GPU 32GB active/kv cache system).
For that price your using a Codex / Claude Pro subscription for the next 4+ years with better models (by default), let alone with a FP2 GLM 5.2 version. And your looking at < 10 fps. A MacStudio with 512GB will net you 18 a 20fps+ with FP4, but ... i mean, those used to be $10.000.
Unfortunately the local hardware cost is a major issue for running large models like that.
Edit: Its funny whenever the issue of cost and what you need to give up vs the subscription services, there are always people who downvote in bad faith.
The cost of local hardware is amortized if a whole team uses it instead of just 1 dev (GPUs are extremely underutilized if you launch just 1 generation stream). I'm not sure why everyone always assumes solo devs with Macs. We've just ordered a large datacenter-grade node for use by the whole dev team, and the calculations show that it's going to cost the same amount of money if we kept using AWS Bedrock (infosec reasons) for a couple years but... it gives us 100% privacy, we're immune to all the AI regulation dramas in the US/EU, all the random outages, and the developers won't have to think about token limits/weekly caps etc. ever again. And all that with a model which is Opus-grade
(it's not our first AI server, we already have experience deploying LLMs for our clients, so the numbers look solid)
Yes but unfortunately a lot of the discussion that people participate in, are not done from a corporate point of view, but from a normal consumer level.
And there is a lot of drama in those discussions. GLM 5.2 is a great model for corporations to run, but people only want to hear about running a 35B/27B or maybe a 120B model. And in that market, subscription services are simply way better value for money (take in account the privacy issues).
Everybody wants GPT 5.5/Opus 4.8 Max levels, on a budget that simply is not realistic. And GLM fit in that 4.8 medium/low level.
But then people do not want to be told that running a 750b model in Q2 or Q1 is just going to destroy the models accuracy. And that is still going to cost them 5k+ for that reduced model.
The whole local llm landscape from a consumer point of view, is just filled with odd people. lol.
Corporation really benefit from those models, because spending $90k on a server, is a deductible expense. And they are billed at token prices anyway from all the major providers. So its a even faster ROI on that hardware.
I am surprised that nobody figured out to make a business of selling leftover capacity from corporate llm installations, because there is easily 12h+ just wasted (unless its a large corp that has people in all timezones).
Some of the inference engines can process multiple requests in parallel more efficiently than doing them sequentially. Not sure of the exact mechanism but e.g. llama.cpp's llama-server can do this (you tell it the number of slots to have when starting, then fire HTTP requests at it and it batches them together when it can).
Waiting for the hooman (or tool calls) won't help either, of course.
The mechanism is that generating tokens (the "decode" phase) in an LLM is limited by memory bandwidth for the weights, so computing multiple streams amortizes the bandwidth over streams as long as you can keep the contexts in RAM. This is most true for dense models and the always-on expert in MoE models, or when you have significantly more streams than experts for MoE models.
In contrast, prompt prefill is more easily compute-bound, so there are interesting trade-offs for latency of decode vs prefill when the LLM utilization is high.
All your going to end up with this type of cases is:
* Years of stress
* Years of financial losses because lawyers are not cheap. And no matter how well you are innocent, not having a lawyer is guaranteed that you will fail.
* Years of time wasting.
And for what? The government maybe sentencing a guy for fraud. When its like 90%+ he will strike a sweetheart deal with the prosecutor.
Even worse in a case like this where its almost your word vs the boss his word. Yea, you can be the guy that ends up living under a bridge while the CEO laughs his way to the bank, being able to pin it on you.
Its already difficult with some proof... Dealing with this type of fraud case reporting, is easier when your not in the spotlight of the crime, and then reporting it. But if your unwilling part of it, few people want their neck on the line.
Neuralwatt ... When you reverse calculate the actual energy usage / price on a token basis, the gap is large.
I do not have GLM 5.2 numbers because the whole default max setting is overkill. But GLM 5.1 numbers had it at 12x cheaper then API rates. And about 2.5x more tokens vs zai their own subscription service.
Yes, its FP8 but lets be honest, do we know for sure that even zai runs at FP16? I learned a long time ago with Claude and Codex how much cheating happens on model levels, even from the big boys.
Please correct me if you have contradicting data but: Neuralwatt's price per token vs price for energy comparison doesn't seem to take into account the cost savings from cache hits that other providers offer on pure token rates. The comparison seems to assume every input token is a cache miss.
On top of that, the cloud offering doesn't seem that well-run, they randomly blocked a colleague's API key for a couple days without any heads up, had a weird rate limiting bug and they have been deprecating models without redirects with very short notice, all while taking weeks to onboard new models. I assume some of these problems would be addressed if we had an SLA/enterprise contract.
It's a promising idea though. They offer a $5 trial credit (with an aggressive rate limit) though so no harm in trying it out.
> doesn't seem to take into account the cost savings from cache hits
Absolute false information.
From my usage panel for this month:
* Total Tokens 1.1B
* Cached Tokens 1.0B 97% of prompt tokens
* Cost energy pricing $26.58
The energy pricing is higher then what i actually pay because its a mix of token billing and partial subscription (60% extra "power").
From the $50 subscription, i have about 3/4 left (4.21 of 16.0 kWh used this billing cycle). Used $5.5 in token billing.
That was running 82.0% GLM 5.1, and 18% GLM 5.2. Yes, i have been busy ;)
My actual usage if we look in dollar value was ~ $18.
For your information, that is cheaper the MiMo v2.5 Pro from Xiaomi as there i was doing around 450.000t per cent. And they have the same 75% cheaper prices like DeepSeek. MiMo has a issue with cache retention between session prompts what hurts them vs DeepSeek. Yes, DeepSeek v4 Pro is 2.5x cheaper but nowhere near GLM 5.1, and especially not GLM 5.2.
In case your wondering, zai subscription light is about 80m token / week limit. So on a token/cent price, neutralwatt is about 3x cheaper (and not 5h, week limits to maximize/frustrate).
> all while taking weeks to onboard new models.
Took them 1 day to include GLM 5.2 ... Yes, the remove old models fast because they do not have the server capacity to keep old models around.
> I assume some of these problems would be addressed if we had an SLA/enterprise contract.
Its a small team, not a big huge company. From my experience so far, seen a 2 timeouts, and sometimes slow speeds as servers get overloaded. For what i am paying for GLM ~5.1~ 5.2 ...
Your reply doesn't seem to be in good faith. Please provide your formula for calculating effective per token cost.
I am not sure why the small team argument is relevant. This is a crowded market, there are dozens if hundreds of third party inference providers in the world right now. I'm glad that's a good excuse that works on you but I'm not sure why the average user should care.
The formula is very easy. Go to the website of neuralwatt, and read ... 5$ = 1Kwh in power for non-subscription usage. For subscription usage you get ~50% more.
Then you actually use the service and see how much tokens you use on average. You calculate the token use vs what you pay. And this gives you a stable number to compare different services and model with, if you want the token cost. This is basic school level reasoning and calculation.
> I am not sure why the small team argument is relevant.
This is relevant to the previous poster his question regarding support and SLA/enterprise support.
> Your reply doesn't seem to be in good faith.... I'm glad that's a good excuse that works on you ...
Question: Do you have a issue with communicating with other people in real life?
GLM 5.2 Max = Opus 4.8 Max in thinking behavior. The thinking chain is so similar, and so is the amount of token usage on the output.
If you want reasonable token usage, you need to run it GLM 5.2 at High. There is little drop in quality from Max to High (for most tasks). And it cuts token usage by 2 a 2.5x. GLM 5.2, Max is really something you only need for complex tasks.
In essence, GLM 5.2 is Opus 4.8 its little brother, at a way, WAY cheaper price.
There has been really no training on Opus models going on, really, none i tell you! /sarcasm
Are there any indications that this will be possible? Consumer hardware will continue getting better but I can't see 512GB RAM in a MacBook Pro any time soon. I'm hoping linear attention techniques plus MoE will make breakthroughs in size/compression and throughput.
> but I can't see 512GB RAM in a MacBook Pro any time soon
Could totally see this being a comment from a forum in like 1994 but swap out GB for MB and MacBook Pro to whatever the popular consumer pc was at the time
Well, we're probably not going to be running frontier models anytime soon, but I think the general assumption is smaller models will continue to improve until they're sufficiently good frontier models aren't needed.
There's potentially also augmentation through tools, harnesses and RAG to help boost how well they work without tons of parameters.
In the last ten years laptop memory footprints have, what, doubled at the low end? Smallest MacBook Pro in 2016 was 8GB, smallest is 16GB today? Max I think has gone up 8x meanwhile, 16 to 128?
I wonder if there's a bit of a chicken-and-egg issue where there wasn't much that demanded 10x the RAM, so there wasn't much pressure to develop more or increase production to support it at consumer prices.
There's wayyyyyyy more demand for memory generally now, so assuming it's not a demand bubble that pops rapidly, I'd expect the new normal to end up at a much higher baseline. 512GB would be 4x greater than today's max, so even with the relatively slow last 10 years development pace, give it five years max?
The problem is that the situation in the RAM market might just... not go away. It's locked in for the next couple of years unless the AI market goes pop. Which it might! But if it doesn't, there's no particular reason to think that the incentives for cornering the market like OpenAI have would go away.
We might see that new normal in five years or so. We will see a new normal sooner than that if there's a run on AI because of the sudden availability of DRR fab capacity, but also we'll probably see the level of local models freeze at whatever state they've got to at that point. But an equally likely outcome is that any new DDR capacity that comes online is just immediately absorbed by frontier AI, and consumer devices stay at "just good enough" for a decade.
The new Macbook Neo is 8GB. I think that if we are lucky, the huge RAM demand right now means new factory buildouts which eventually means more supply and prices go back down, and capacity begins to go up. This level of demand was just not anticipated by anyone.
so around US$150k which is Small/Medium-Enterprise territory already, but who knows when it will hit "reasonable" home consumer territory
I think there's hope future generations of unified memory machines may get this sort of memory availability when new fabs open in then next couple of years and then ramp up production for a few years afterwards - that makes ~2030s credible at this point, but nobody can really predict the market that far ahead
> I think there's hope future generations of unified memory machines may get this sort of memory availability
I hope you're right. This is a very exciting idea. The weights are out there. The demand is astronomical. The manufacturers just need to make it happen.
there are cheaper ways to do it. not like, consumer-cheap, but I'm setting up a rig for 80% cheaper than that.
I'm a tad worried about triggering a run on the particular hardware I'm buying though so I'll leave it vague here, but hit me up on Discord if you're curious.
This is quite evident for personal AI but general intelligence with current scaling laws and how model keep getting better with more number of parameters, certainly the path does not converge.
Personal AI is more deprived of context today than quality of token. Having a on-system knowledge base paired with Gemma works well to large extend.
With such ridiculously long thinking traces I'm surprised max outperforms high. After all, performance falls off a hill after a certain amount of context, and long thinking traces can fill that up really quickly.
distillation of thinking models is not particularly effective - both "Open"AI and Misanthropic don't show you the real chain of thought, only its severely downscaled version. both do everything in their power to combat such outrageous copyright infringement, so the bulk of unethically scrapped data the Chinese have is from several generations ago.
It is quite likely that the intermediate tokens don’t have ‘semantic import’[0]
There are methods like Habitual Reasoning Distillation or Inverted Reasoning Traces [1] that can help.
While there are reasons to hide the intermediate tokens from a IP protection stand point, there is also a need to hide more effective and efficient generating that doesn’t fit the R1 claims of an aha moment that has been debunked, but is a consumer expectation.
While hidden intermediate tokens do increase the difficulty, it is not a from barrier in itself, especially as they are billed, given information about their length.
Chinese distillation attacks are about as unethical as Robin Hood stealing from the rich to give to the poor. The real unethical scraping was done by Anthropic to train Claude.
To be clear, if Anthropic was using totally licensed data, I'd be sympathetic to these claims. But if you're going to pirate the world's creativity you'd better be willing to gimme dat shit for free[0].
For Claude models at least, you can tell to just manually think in the output and it works fine. I do it reguralrly because for creative writing and summarization, they seem to believe they don't need to think at all, and get way worse results.
this helps so much. i do it too. with some of the newer frontier models its unclear if you can even turn it off in the first party chat apps. havent compared api semantics yet.
The companies that did copyright infringement and unethically scrapped data think that copyright infringement and unethically scrapping data is wrong and needs to be stopped.
Though only in particular situations, like when it’s done to them and not when they do it. Cause they have the power and are morally right and know better than you. And if you question this at all, well you’re a threat to American values and a supporter of the Chinese and leading to the break down of Democracy.
This isn’t a type of reasoning argument or manipulation tactic used by the rich throughout history to trick the naive and gullible masses or anything like that. Trust me, I’m rich and I’m morally right. /sarcasm
It’s been amazing to see the arc of tech people going from “evil Disney, copyright is an abomination, information wants to be free” to “OMG copyright is inviolable and AI is taking money out of Plato’s descendants’ pockets!”
> taking money out of Plato’s descendants’ pockets
Yeah, remind me - is it Plato's descendants that people are concerned about here, or is it every single author who had any work in Anna's Archive, any work published online, any work published on github, etc?
I think that people are probably upset about the harm to living people who had their work stolen by Meta and other LLM companies - regardless of license, terms of use, or any other attempted protection.
Sure, that’s the motte / bailey. Easy to point to living, starving writers who suffer grevious harm, in defense of perpetual copyright. Disney and others use literally this exact argument year after year.
I’m not even disagreeing. I’m just saying the shift in attitude about copyright in the tech space has been sudden, dramatic, and really funny. Remember “you wouldn’t steal a car”? Today’s anti-AI tech contingent are enthusiastically embracing that false equivalence that we all laughed at 20 years ago.
Having a static, immovable belief system about something like copyright that is unaffected by seismic shifts in the real world also doesn't seem very logical.
If like, Disney did a 180 overnight and bought rights from Google to scan every writer's saved work in Docs with some flimsy legal argument then a person saying "wait doesn't copyright actually protect that" would make sense. Even if you were previously upset about them suing schools for using 80 year art.
Simple trick: Use an agentic tool like Pi or OpenCode that allows you to switch models. First do some chats with DeepSeek or GLM who shows full thinking traces, then switch to Claude or GPT and it's more likely to show full thinking traces.
I don’t understand why there isn’t public dataset for reasoning that can be improved by humans/llms like Wikipedia (ie with auto judging contributions etc).
For reasoning a manually-curated dataset is too small; you need to be able to automatically generate vast volumes of synthetic reasoning data with provably correct answers. That's presumably why Claude and GPT are so good at using Lean (the theorem prover), because they get fed a bunch of synthetic, verifiably correct training data.
They are scaling up, but most will only come online in end 2027-2028 time frame. And Memory, as in what we use in PCs is easier to manufacture then HBM memory. But all the money is in HBM ...
So for every ~4GB of memory that you can produce in normal DDR5, you can only make 1GB of HBM. But you make multiple times the revenue.
The demand for HBM memory is not going to go away. LLMs are memory bandwidth hungry, and we are going to see production going to AI. But also to "lower end" like B200's.
That means, they are producing multiple times less memory (if we look for the normal market demand), but still need to produce more for the memory bandwidth hungry market.
We are seeing more products entering the "prosumer/business" market that are also memory bandwidth hungry. This demand will not go away. It will actually increase as companies move to more localized workloads. There is is a issue with data privacy that a lot of companies legally deal with.
The lacking ramp up is not a sign of them being scared of over production, its a realization that 3 companies hold the market in a strangle hold, and "slow" scale. If everybody plays friendly, they can milk this for years.
China is a solution but China does not have the HBM production levels, and will take years to scale and put a dent in the market. And China is ... allocating a lot to domestic production of AI > HBM ...
The reality is, that unless competition ( as in China ) does not start scaling beyond the expected levels, the big 3 have no reason to scale too fast.
And money is not the issue ... have you seen their revenue (and net profit!! ) numbers. A few billions is peanuts for them at this point. They simply do not want to scale too fast because that means less milking ... Memory demand is not going to away. When people talk about the AI bubble popping, its more in terms of the stock market. The product is here and not going away.
This does suggest a path to improvement, though. A significant factor in the demand for HBM is how expensive the actual GPU chips are, making you want to use the absolute best memory to support them. When there's more competition in GPUs and the memory is actually a lot of the total price, you see things like Apple silicon with LPDDR5 being very popular. You can get a lot of bandwidth out of normal memory if you put in 256 or 512 bit bus. If we can get more midrange competition, we can focus more manufacturing capacity back on some form of DDR, and lessen the squeeze.
These prices have absolute nothing to do anymore with memory prices. Do not forget that Hetzner already increased the setup fees by a factor of 4x before to compensate for the price. And also servers getting price increases.
It seems they have shifted by reducing the setup fees, and increasing the monthly costs. As this generates more revenue. And its easy to prove this...
AX42 ... Its 8700GE that has gone from 65 Euro to 225 Euro. With the setup fee now being 112 Euro instead of 225 Euro. It has 64GB memory, and 1TB storage. The storage even in todays market is 100 Euro. The memory is 644 Euro.
Do the math ... Hetzner servers had a hardware payback periode of between 9 to 11 month if you took the market value. This calculation has always been very stable over the 20 years i used Hetzner.
This new price, reduced the hardware payback periode to ~4 month. It seems to be that Hetzer is trying to use the memory price issues, as a excuse. The revenue of those same servers now increased to a insane level. More revenue with less hardware.
The real issue is that a lot of companies are moving from US hosting to EU hosting because of the problems with the US. Hetzner sees this as the perfect time to cash in on Enterprise customers.
They have been trying to replace the "cheap" normal consumers with enterprise. This trend has been going on for a while already.
Every customer that now leaves, is a server they can rent out to business customers.
If you want to see the same thing, look up what happened to Microsoft/Github Copilot where they turn around has been sudden and very strong, with a clear goal of moving everything to enterprise.
The big increases here are for their cloud product, which is hourly billing with no setup. In that context it seems more reasonable. I guess we need to remember that hourly billing and flexible prices cut both ways, eh?
> It seems they have shifted by reducing the setup fees, and increasing the monthly costs.
Monthly costs have gone up as well. Payroll has seen significant increases in Germany, construction has exploded far beyond inflation and, most importantly, electricity prices are still ridiculous due to merit-order and the refusal of splitting up Germany into multiple power pricing regions.
I remember the price increase that Hetzner did during 2022 because of the invasion in Ukraine. The said they will adjust the prices down when the electricity price reduced.
Guess what? I am paying as a consumer about the same price as before 2022. Did Hetzner change their price down? Remember, the industrial price also dropped (and they also build out a large solar plant). No ...
Ok, inflation? But those price increases already covered part of that... Just saying, its not been the first price increase that happened. There have been multiple ones that Hetzner did over the years. Some flew under people radars.
> Payroll has seen significant increases in Germany,
Yea, we have seen nothing of that increase... O, wait, they reduce our income because the social security increase their costs. Yay ..
Since I started using Hetzner in 2020 they have increased prices 4 or 5 times and I am now paying 50% more than I started, but my grocery bill also went up 50% and my rent went up 50%, so that's just matching inflation (even though the government said inflation was 5%). Now they're doing a 300% increase (not for me) all at once.
The reasons really doesn't matter. As long as they are in top of price/performance/quality nobody will switch. Once they stop to be then people will think about it.
If the rest of the market stays at current prices sure. It's very likely that Hetzner is just popular and ran out of hw quicker than others. I doubt they changed their market strategy, they are probably just being realistic. We will see if/when the hikes come to others.
What sucks is that it might just be that hyperscalers have long term hw contracts and completely starve any competition.
Also, a price increase like this can be used to address over-subscription/under-utilization .. there will be a lot of dormant chaff blown off by this, or in other words the provisioning demand will also be adjusted by this aggressive price change, imho.
> Only U.S. citizens and immigrants that are holders of a "green card" may now access Mythos.
It was my understanding that not even green card holders may access Mythos. Normally when restrictions like this are put in place, you need a exemption as a green card holder. A geen card is just a permit to live and work in the US. Its not the same as citizenship.
> Security Clearance: Green Card holders are generally prohibited from jobs that require high-level security clearances or sensitive government/military roles reserved exclusively for U.S. citizens.
That's not true. ITAR and security clearance are entirely separate regulatory regimes. For ITAR purposes, being a permanent resident is good enough. I used to work for a defense contractor, and we hired plenty of green card holders. They were not in general assigned to work that required a clearance, but plenty of defense-sensitive, ITAR-controlled work is done by green card holders.
> I don’t understand how I grew up thinking USA is the gold standard is good and China just make cheap copies and is bad
You did not grow up in the 80s ... Where it was the same about US vs Japan. Look how it turned out for several of the US industries. The US tends to sleep, look down on other countries, and then it loses key industries because of that attitude.
It’s not just the US. Honda recently announced they’re not able to make internationally competitive (=== high quality and affordable) cars anymore, and abandoned their entire EV line.
I guess they’ll just milk the ICE assembly lines until they are bailed our or go under, Detroit-style.
It did not stop solar panels getting cheaper and cheaper because of the whole integration and mass production (with healthy free market competition).
The last subsidies like export value-added tax rebates for solar panels and lower rebates for batteries are ending in 2027.
China their main power is, the ability to have everything inhouse. Yea, they subsidize a lot of stuff until it hits critical mass, and then you have often a healthy industry with lots of competition.
China alone has like a few 100 car manufactures because of the subsidies, and over time there will be consolidation / buyouts etc but the end result is a healthy new industry that exports. With again, everything internally being produced.
This is why our subsidies fail. We do one sector, often a few companies at best. This results in few competitors, expensive prices, and often reliance on externals that can bankrupt those companies. And que how we wasted again dozens of billions in propping up a industry with no competitive edge.
People can cry about China but they are actually doing work, despite the mass amount of corruption. That is the big difference with here... Mass corruption got in the way of national security, plop, people go to jail. Industry quickly gets their ** together. Here ... give billions, and the money vanishes, with no real consequences.
reply