Cost wise it does not seem very effective. .5 token / sec (the optimized one) is 3600 tokens an hour, which costs about 200-300 watts for an active 3090+system. Running 3600 tokens on open router @.4$ for llama 3.1 (3.3 costs less), is about $0,00144. That money buys you about 2-3 watts (in the Netherlands).
Great achievement for privacy inference nonetheless.
Something to consider is that input tokens have a cost too. They are typically processed much faster than output tokens. If you have long conversations then input tokens will end up being a significant part of the cost.
But why not cross that bridge then. By that time you might have much more optimized local infrastructure. Although I do see that someone suffering through the local slowness now is what drives the development of these local options.
Why is this so damn important? Isn't it more important to end up with the best result?
I (in Norway) use a homelab with Ollama to generate a report every morning. It's slow, but it runs between 5-6 am, energy prices are at a low, and it doesn't matter if it takes 5 or 50 minutes.
author here! i originally intended this to be a technical write-up on everything i learned building an ATS (as someone who's in HR). the original title was "how to build an ATS and why you probably shouldn't". i debated mentioning what i built, but it felt hollow to _ramble_ about schemas, architecture, components with nothing to show for it.
the more i wrote and reflected, the more i thought about why the market never corrects itself despite the tools being expensive and badly designed. i've worked with hundreds of recruiters and most use spreadsheets. that's not a workflow quirk but i think an indictment of something bigger which traces back to everything in the post -- the buyer who never uses the product, the integrations racket, the "AI-native" tools bolted on top of a broken foundation, etc. etc.
so i ended up writing the first half. it's drawn from my experiences frustratingly buying an ATS for a small business, and watching the dysfunction of procurement/integration/lack of adoption play out at the enterprise level.
admittedly, HR/recruiting tech is a very niche audience, so the technical section probably lands better with engineers who've been handed a recruiting project than with anyone actually working in HR. so i wanted to offer a resource from that perspective.
Surprising how badly Jetbrains implemented AI. Apparently to such an extent that even after multiple years of LLM's someone felt confident enough to build a company that can do better.
This looks really neat, interesting technical writeup as well!
Cool social experiment. It's interesting how narrow the scope of all top voted PRs are: change this or that detail in the voting (daily, count down votes etc), or make it more efficient (rust).
I wonder if this has the potential to build a "community" that will take this into a completely different direction, or if it will neatly stay within the initial boundaries.
I think if the end game is to run workers runtime then they could also run something else from the start.
Its gonna be hard to compete with the scaling cloudflare offers if they migrate to their own dedicated infra, but it of course would become much cheaper than paying per request
Smart 11 | React Native Software Engineer | Netherlands (Breda) / Remote | Part or Full time | smart11.ai
Smart11 helps soccer players to become more intelligent on the field. We're a small team of 3 engineers and are looking for a react native developer to continue development on our mobile app.
Our soccer learning methods rely heavily on video (and a touch of vision AI) to help players learn from their own actions on the field. The app is crucial in that process, and used hundreds of hours a week by clubs and pro players alike. Your work might directly contribute to the improvement of soccer players around the world!
We're a productive, balanced team, and are looking for someone with a builder mentality. Apply at connect [at] smart11.ai, or send me a PM (I'm the CTO). If you're interested but are only experienced in related techniques, please do get in touch, we like to invest in the right people!
Super easy? Getting that setup for the first time from zero knowledge will probably take a few days. And that's before understanding all the intricacies like your AWS bill, the hidden costs of cloud, properly securing IAM and setting up your VPC etc etc
Not once in my entire career have I seen people successfully avail themselves of the cloud's purported benefits. I have, however, seen a lot of happy account managers in Las Vegas during Re:Invent, oh yes.
All the Redis users have is a license to use and an expectation. An expectation is a belief that Santa will bring presents, that's all.
Where the value is or was is pure sophistry. You don't have a crystal ball, just like everyone else.
All this discussion is envious bellyaching from those that are probably leeches themselves. They just want the free gravy train running for themselves.
And the license allows them to fork it. Which is what they are doing. Open Source working exactly as it should. I just want to be sure the facts are understood. Amazon has many faults and there are plenty of reasons to dislike and not use them. But leeching off of Redis Labs is not one of them.
From my point of view managed databases only really make sense for toy projects, if you’re using these things at scale it’s much more economical to buy some servers and hire some people of your own, and use plain pre-VC Redis. But big corporations seem to have some kind of a fetish for lighting money on fire, and the fight here is fundamentally over in whose fireplace to do it.
> From my point of view managed databases only really make sense for toy projects
it is more expensive to buy managed, but you offload work. I would imagine toy projects are more cash constrained, and makes more sense to rent cheap servers and roll your own.
On the other hand, larger scale projects would rather pay to offload the work of managing and scaling redis.
Toy project are both cash and time constrained, but they’re at a scale where managed is cheap enough because they want to get you hooked.
Large scale projects can take advantage of economies of scale and hire ops people. I’ve found cloud support pretty lacklustre compared to having someone to talk to face-to-face who understands the whole stack for your particular application.
Of course conventional corporate wisdom says waste as much as you like on services as long as you keep payroll down, that may be a bigger challenge than any of the technical ones.
In my experience using redis, one of its better attributes is how easy it is to manage and scale. I've never scaled it to say, Facebook levels, but at that scale, I'm not sure managed services make much sense either.
Yes, it is ludicrous. My company uses hosted databases and "droplets" from DigitalOcean. Their pricing is absolutely absurd. I always wondered how they stay in business, but now I know.
I'm using it for a 10.000+ multisite installation (quite simple personal blogs), and even though it has it quirks it works quite well. Almost nothing is best in class and a slight pain to get it working, but the sheer size of the ecosystem is still it's biggest plus. In general every problem or feature you can imagine already exists or is easy to bolt on.
Great achievement for privacy inference nonetheless.
reply