I find it quite funny that they give themselves maximum marks for transparent pricing. If you go to their pricing page, everything is priced as “per user/month plus compute costs*”. Maybe it’s just because I’m on mobile and the page doesn’t seem to work super well, but reading if I have no idea what those compute costs are and therefore what the actual cost is.
Honestly, this is super fair. I went in expecting to hate-read it (not because I have any issues with Render, they're great, just these X vs. Y competitive posts). But yeah, I think this is a reasonable way to look at it.
I have been using Google voice for 10+ years, and it's always been something that concerns me. I've been worried about Google either banning me or dropping the service. Does anyone know of a good alternative, I'm happy to pay for something but the main thing is it needs to be able to work from the browser like Google voice.
I've been pondering something based on twilio or similar, but haven't found anything. Don't really need the web calling, but web sms would be useful at times.
jmp.chat doesn't really fill me with confidence, but it might be an option in an emergency.
Shameless plug: This is a podcast interview I did with Yi Ding of Llama index where we discussed some of the differences between LangChain and LlamaIndex (among other things). Give it a listen as a supplement to the article:
I can share a few things! (but definitely not all)
- One is finding different T7 RNA polymerases with unique properties by manipulating the backbone. They can be used for things like in-vitro RNA production for vaccines
- Another is synthesizing a phage that has been sequenced for a specific organism, but that the samples are now lost of. So resynthesizing that genome from scratch
- A different project (personal one) is building a DNA parts toolkit with standardized DNA parts so you can combine em together like legos. Pretty much nowhere but FreeGenes has open source genetic parts (I used to run that project), and I think open source genetic parts need to be in the world
Somewhat related is "Oh My ZSH!" which is basically zsh on steroids, it's always one of the first things I install on a new computer. It gives things like new colors, themes, plugins, and more. Highly recommend you check it out.
I used OhMyZsh! for a long time and really liked it, but now I moved to Starship [1]. However I noticed I still missed some of the "steroids" that OhMyZsh brought so I keep it installed in my system and I only source the plugins I need in the .zshrc:
A Studio Mac with an M1 Ultra (about 2800 USD used) is actually a really cost effective way to run in. Its total system power consumption is really low, even spitting out tokens at full tilt (<250W).
You can run a similarly sized model - Llama 2 70B - at the 'Q4_K_M' quantisation level, with 44 GB of memory [1]. So you can just about fit it on 2x RTX 3090 (which you can buy, used, for around $1100 each)
Of course, you can buy quite a lot of hosted model API access or cloud GPU time for that money.
RTX 3090 has 24GB of memory, a quantized llama70b takes around 60GB of memory. You can offload a few layers on the gpu, but most of them will run on the CPU with terrible speeds.
* Turing / Quadro RTX / GTX 16xx / RTX 20XX / Volta / Tesla
EOL 2023/2024
=============
* Pascal / Quadro P / Geforce GTX 10XX / Tesla
Unsupported
===========
* Maxwell
* Kepler
* Fermi
* Tesla (yes, this one pops up over and over, chaotically)
* Curie
Older don't really do GPGPU much. The older cards are also quite slow relative to modern ones! A lot of the ancient workstation cards can run big models cheaply, but (1) with incredible software complexity (2) very slowly, even relative to modern CPUs.
Blender rendering very much isn't ML, but it is a nice, standardized benchmark:
As a point of reference: A P40 has a score of 774 for Blender rendering, and a 4090 has 11,321. There are CPUs ($$$) in the 2000 mark, so about dual P40. It's hard for me to justify a P40-style GPU over something like a 4060Ti 16GB (3800), an Arc a770 16GB (1900), or a 7600XT 16GB (1300). They cost more, but the speed difference is nontrivial, as is the compatibility difference and support life. A lot of work is going into making modern Intel / AMD GPUs supported, while ancient ones are being deprecated.
P40 is essentially a faster 1080 with 24GB ram. For many tasks (including LLMs) it's easy to be memory bandwidth bottlenecked and if you are they are more evenly matched. (newer hardware has more bandwidth, sure but not in a cost proportional manner).
I find that my hosts using 9x P40 do inference on 70b models MUCH MUCH faster than a e.g. a dual 7763 and cost a lot less. ... and can also support 200B parameter models!
For the price of a single 4090, which doesn't have enough ram to run anything I'm interested in, I can have slower cards which have cumulatively 15 times the memory and cumulatively 3.5 times the memory bandwidth.
Technically, P40 is rated at an impressive 347.1GB/sec memory bandwidth, and 4060, at a slightly lower 272GB/sec. For bandwidth-limited workloads, the P40 still wins.
The 4090 is about 3-4x that, but as you point out, is not cost-competitive.
What do you use to fit 9x P40 cards in one machine, supply them with 2-3kW of power, and keep them cooled? Best I've found are older rackmount servers, and the ones I was looking at stoped short of that.
I technically have 10 plus a 100GBE card in them, but due to nvidia chipset limitations using more than 9 in a single task is a pain. (Also, IIRC one of the slots is only 8x in my chassis)
Supermicro has made a couple 5U chassis that will take 10x double width cards and provide adequate power and cooling. SYS-521GE-TNRT is one such example (I'm not sure off the top of my head which mine are, they're not labeled on the chassis, but they may be that).
They pricey new but they show up on ebay for 1-2k. The last ones I bought I paid $1800, I think the earlier set I paid $1500/ea-- around the time that ethereum gpu mining ended ( I haven no clue why someone was using chassis like these for gpu mining, but I'm glad to have benefited! ).
P40 still works with 12.2 at the moment. I used to use K80s (which I think I paid like $50 for!) which turned into a huge mess to deal with older libraries, especially since essentially all ML stuff is on a crazy upgrade cadence with everything constantly breaking even without having to deal with orphaned old software.
You can get gpu server chassis that have 10 pci-slots too! for around $2k on ebay. But note that there is a hardware limitation on the PCI-E cards such that each card can only directly communicate with 8 others at a time. Beware, they're LOUD even by the standards of sever hardware.
Oh also the nvidia tesla power connectors have cpu-connector like polarity instead of pci-e, so at least in my chassis I needed to adapt them.
Also keep in mind that if you aren't using a special gpu chassis, the tesla cards don't have fans, so you have to provide cooling.
I don't remember exactly (either cuda directly or the cudnn version used by the flashattention)... Anyway, /r/localLlama has few instances of such builds. Might be really worthwhile looking that up before buying
The answer is the majority of the Republican party and their electorate, but just saying that will get you a barrage of downvotes on HN and other similar forums.
It was top billing in Democrat policy for decades, from Clinton through Clinton, and Biden did some too. Top billing in Republican policy was to oppose it and undermine it - remember the endless, pointless, virtue-signaling votes during Obama's administration by Congressional Republicans to kill Obama's healthcare plan. Remember he got it through on the slimist margin, through procedural maneuver, with (no? one?) Republican votes.
Organize and get the ball rolling. What are the problems? How do we solve them? You're not a customer or a victim; you tell them what they should be doing. Don't worry, most politicians will do anything to get some votes!
I remember how the ACA was essentially a giant gift to the insurance industry, and that despite being mandated in most states, only increased the insurance rate by a few percent. Oh, and the vast majority of the increase in insurance was health plans with a $10000+ deductible. I remember how medical debt bankruptcies continued to climb after it was passed.
> Don't worry, most politicians will do anything to get some votes!
Before they get votes, they have to win the money primary. Then we get to decide which technocrat or self-funded billionaire candidate sounds nicest to our ear.
It’s time to stop thinking you can vote your way out of this. Our leverage is our labor.
A few alternatives to consider
- https://render.com/ - this is very close to heroku
- https://coolify.io/ - My personal favorite. It's slightly more involved, but you can run it on any hardware like hetzner and save a boatload.