There has been a lot of opinion pieces popping up on HN recently that describe the benefits they see from LLMs and rebut the drawbacks most of them talk about. While they do bring up interesting points, NONE of them have even mentioned the privacy aspect.
This is the main reason I can’t use any LLM agents or post any portion of my code into a prompt window at work. We have NDAs and government regulations (like ITAR) we’d be breaking if any code left our servers.
This just proves the point. Until these tools are local, privacy will be an Achilles heal for LLMs.
Trivial after a substantial hardware investment and installation, configuration, testing, benchmarking, tweaking, hardening, benchmarking again, new models come out so more tweaking and benchmarking and tweaking again, all while slamming your head against the wall dealing with the mediocre documentation surrounding all hardware and software components you're trying to deploy.
Even my 4-year-old M1 Pro can run a quantized Deepseek R1 pretty well. Sure, full-scale productizing these models is hard work (and the average "just-make-shovels" startups are failing hard at this), but we'll 100% get there in the next 1-2 years.
Yes. The past two companies I've been at have self-hosted enterprise LLMs running on their own servers and connected to internal documentation. There is also Azure Cloud for Gov and other similar privacy-first ways of doing this.
But also, running LLMs locally is easy. I don't know what goes into hosting them, as a service for your org, but just getting an LLM running locally is a straightforward 30-minute task.
Run a vllm docker container. Yeah the assumption is you already know what hardware you need or you already have it on prem. Assuming this is ITAR stuff, you must be self hosting everything.
Yes, but which of the state of the art models that offer the best results, are you allowed to do this with? As far as I've seen the models that you can host locally are not the ones being praised left and right in these articles. My company actually allows people to use a hosted version of Microsoft copilot, but most people don't because it's still not that much of a productivity boost (if any).
Deepseek isn't good enough? You need a beefy GPU cluster but I bet it would be fine until the large llama is better at coding, and I'm certain there will be other large models for LLM. Now if there's some new technology around the corner, someone might be able to build a moat, but in a surprising twist, Facebook did us all a favor by releasing their weights back when; there's no moat possible, in my estimation, with LLMs as it stands today. Not even "multi-model" implementations. Which I have at home, too.
Say oai implements something that makes their service 2x better. Just using it for a while should give people who live and breathe this stuff enough information to tease out how to implement something like it, and eventually it'll make it into the local-only applications, and models.
I can fill in anything missing, i would like to go to bed but i did't want to leave anyone hanging. had to come edit a comment i made from my phone, and my phone also doesn't show me replies (i use materialistic, is there a better app?)
https://news.ycombinator.com/item?id=42897205 posted here, a link to how to set it up on an AMD Epyc machine, ~$2000. IIRC a few of the comments discuss how many GPUs you'd need (a lot of the 80GB GPUs, 12-16 i think), plus the mainboards and PSUs and things. however to just run the largest deepseek you merely need memory to hold the model and the context, plus ~10% and i forget why +10% but that's my hedge to be more accurate.
note: i have not checked if LM Studio can run the large deepseek model; i can't fathom a reason it couldn't, at least on the Epyc CPU only build.
note too: I just asked in their discord and it appears "any GGUF model will load if you have the memory for it" - "GGUF" is like the format the model is in. Someone will take whatever format mistral or facebook or whoever publishes and convert it to GGUF format, and from there, someone will start to quantize the models into smaller files (with less ability) as GGUF.
That's $2000 but for just 3.5-4.25 tokens/s? I'm hesitant to say that 4 tokens/s is useless, but that is a tremendous downgrade (although perhaps some smaller model would be usable)
right, but that is CPU only, there's no "tensor cores" in a GPU getting lit up for that 4t/s. So minimum to actually run deepseek is $2000, and the max is, well it's basically whatever you can afford, based on your needs. if you're only running single prompts at any given time, you only need the number of GPUs that will fit the model plus the context (as i mentioned), at minimum your outlay is going to be on the order of $130,000 in just GPUs.
If i can find it later, as i couldn't find it last night when i replied, there is an article that explains how to start adding consumer GPUs or even 1-2 Nvidia A100 80GB GPUs to the epyc build, to speed that up. I have a vague recollection that can get you up to 20t/s or thereabouts, but don't quote me on that, it's been a while.
It is not at all trivial for an organization that may be doing everything on the cloud to locally set up the necessary hardware and ensure proper networking and security to that LLM running on said hardware.
> NONE of them have even mentioned the privacy aspect
because the privacy aspect has nothing to do with LLMs and everything to do with relying on cloud providers. HN users have been vocal about that since long before LLMs existed.
I grew up in a megachurch but I wouldn’t call it a “megachurch”. It (Manchester UMC) is large at about 2500 members but still does the services in the same format as it has been doing since as early as I can remember. However, I have been to a couple of “megachurches” with that style of service and can comment on the appeal.
Services at a megachurch have almost a laid back rock concert type atmosphere. There is a lot of contemporary music played and everything else is made to be as “soulful” and engaging as possible. It’s a very unique atmosphere, but the appeal is much the same as something like a country music concert. The services are a whole production.
The other advantage big churches have (and I did see this with my church growing up) is that larger communities make a lot of things easier. It’s easier to form service missions, bible groups, youth activities, etc, because it’s much easier to reach a critical mass when you have such a large congregation to pull from.
You don’t need to track people to pay for things with ads. You can advertise based on the surrounding content pretty effectively. It’s what TV channels have done for years.
I just used chatGPT to help me write some Powershell scripts. Here’s the kicker though. I have never wrote a Powershell script before and I have only read some on the basics of functions. However, I know enough about programming in general to still tell where the problem could be. It’s been nice so far. It’s never gotten it right the first time but coding is always an iterative process. We did get there much quicker than I would have alone. It essentially taught me Powershell.
One takeaway though is that (at it current level) I still don’t think it will replace programmers. It’s initial solutions sometimes go in the wrong direction, but because I could still understand the code it wrote, I was able to get it on the right course pretty quickly. It often went like: hey i don’t think you should be technique/structure X could you replace it with Y, and it would often get it much closer after that with some minor bugs to fix.
Overall it actually felt a lot like pair programming with someone who knows all the documentation but not always the best way to approach the problem.
I feel like I can offer a unique perspective on this because I go to Notre Dame and we are just finishing the first week of classes with everyone back on campus. It's gone alright so far. All the cases so far have been traced back to off campus bars or parties. However, I expect it to get worse soon as cases start spreading on campus. If you want to see how Notre Dame is doing with cases and testing we have a dashboard here: https://here.nd.edu/our-approach/dashboard/
I really hope we can make this work because, as an electrical engineer, most of my classes can't be moved online.
Yea pretty much. However, I would extend that to off campus students as well because as soon as someone has a really severe case (or dies) then we will probably all be sent home.
Also it's not just bars or parties. You also have to worry about other gatherings. If people don't where masks and social distance those can become a problem as well. However, so far people have been mostly responsible with other on campus gatherings.
Technically it could work if they keep distance and wear masks, but as it looks like this has proven to be pretty difficult for people aged 20-30 so far, mostly because the impact of COVID for them is fairly low.
Or it may work because there’s close to zero chance of dying for college students, and we have far more immune people now than we did back in the spring. Sweden’s numbers are looking very interesting right now.
College students can easily spread it to vulnerable populations.
1. Some live at home.
2. Some will go shopping. Some in places without mask mandates.
3. Older professors and staff (custodial, food services, advisors) will be pressured to work (you probably can't just say "I'm not teaching" without quitting).
4. Some may need to use public transit to get to the school. Again also in some places without mask mandates.
5. And of course, some students are vulnerable. Many schools are not allowing remote work when they open.
Add in dorms being super spreader environments and you've got disasters waiting to happen.
There are a lot fewer people to spread it to now than there was in March.
The USA is over 5M confirmed cases now. The CDC estimates that the true infection count is 6x-24x that. So on the order of 30M - 120M Americans have already had it now.
The distribution is uneven. Places that hadn't gotten hit are getting hit now. But on average we are likely close to herd immunity.
This is no longer an exponential thing here. We can't have an OOM more cases. It's mathematically impossible. The worst is over (again, on average). I don't think we explicitly tried to follow a herd immunity strategy but that is what ended up happening.
but is that reason good enough to take away everyone's right to go to college?
here is where personal responsibility comes into play. If you are concerned stay home. The university and laws should support you.
But the solution is not to punish everyone else, that won't work long term. The sooner people realize that the better. Limiting and collectively taking away people's rights will cause far more trouble long term.
Everybody at every college is saying that it could work if the students behave themselves when they're not in class. Then in the next breath, they say: It can't work.
Oddly enough I went to ND for grad school, though it was many years ago. The campus was isolated from the surrounding town, but not completely isolated.
The numbers, though very few, seem to fit an exponential well, at least thus far.
If you plot and fit the curve to such an exponential, today you should have ~15 cases. Tomorrow ~25 cases. In one week's time, ~380 cases. ND has ~8600 people attending. To get that many cases should be 14 days from today [0], so August 28th.
If these trends continue as is, then ND won't make it to September before closing down.
Still, the data is sparse.
[0] Day 20 is ~6000 cases, day 21 is ~9400 cases, you can round however you'd like.
This just points to the absurdity of the whole situation. I mean in what world should bars be open and universities closed? It’s a complete failure of priorities.
We should be spending the entire defense budget on paying for people to stay home, stay safe, and to get them what they need; as well as funding as much scientific research as feasible to fight this clear and present danger to our lives and livelihoods.
As we saw in Italy and New York, COVID-19 has the potential to overwhelm hospitals. Officially, almost 170,000 Americans have died. NYTimes estimates that COVID-19 already caused over 200,000 deaths by comparing against the expected number of deaths if we weren't in a pandemic. So yes, COVID-19 is a clear and present danger to our lives.
Even the very highest estimates of asymptomatic infection don't put it anywhere near that high, unless you're talking about only the first few days or some such.
Sorry, I meant the traditional definition of "conservative" with regard to media, which is to say ensuring that only unbiased facts are reported as best as possible.
I grew up in South Bend, and most of the 'Fighting Irish' near campus bars are way too small for social distancing. Pre-covid, all of them were shoulder to shoulder packed with students and locals. Please be careful celnardur. It's a wonderful place and campus, enjoy your time there.
Although not exactly an online course, my compilers professor released his book for free online. It walks you through all the steps of building a compiler and is an invaluable resource for me. You can find it at www.compilerbook.org
This reminds me of some of the exercises I did during marching band and Drum Corps. We were taught, like the article says, to breathe by expanding the abs and then use the abs and diaphragm to push the air out through the instrument. In both, we used breathing gym techniques developed by Sam Pilafian and Patrick Sheridan. Here's a video of them demonstrating some of the exercises we did. https://www.youtube.com/watch?v=qEz0ku-oXM4
These exercises definitely helped me prepare my lungs for the stress of what is essentially running and playing a brass instrument at the same time for ten minutes. Even though they are meant for Marching Band and Drum Corps, I would still recommend breathing gym for anyone who wants to be a stronger breather.
In my compilers class we used a book that my professor wrote that takes you through all the steps and algorithms for making a compiler for a c like language. You can find it online for free at:
https://www3.nd.edu/~dthain/compilerbook/
I really like this book because it really takes you through all the steps. However, it's very readable and provides great examples of how to actually implement some of the components.
I also like it because it explains how to use tools like Yacc and Bison, while explaining how they work underneath and the motivation for using such tools.
I leaned on this book heavily in the class where the main assignment was writing our own compiler from scratch in C, and I'm currently using it now to make a compiler in rust for a custom language.
I'm in a college club that uses Pis to run football playing robots. The Pi basically translates instructions from a Bluetooth PS4 controller into commands for a motor driver.
We currently 4 college clubs in the league and we play a couple of scrimmages and a championship every year. The robots take big hits in the games(these bots can go faster than we can run) and we've found the Arduino to be the most durable and reliable platform as well as being the most customizable.
Here is our club Website: https://www3.nd.edu/~rfc/
Here is a little video about the club: https://www.greatbigstory.com/stories/nbc-sports-intramural-...
This is the main reason I can’t use any LLM agents or post any portion of my code into a prompt window at work. We have NDAs and government regulations (like ITAR) we’d be breaking if any code left our servers.
This just proves the point. Until these tools are local, privacy will be an Achilles heal for LLMs.