2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup.
An hour later, he calls back. Hacked again. Restored again.
An hour later, he calls back. He realizes that the hacker is him! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root:
This is precisely why I always `-v` when `rm`ing recursively. It might be closing the barn door after the proverbial horse has bolted; but at least the fuck up is visible and in some circumstances you have a fighting chance to kill `rm` before too much damage has been done.
I neither use -f "all the damn time" and nor do I appreciate that baseless accusation. Furthermore -f has literally nothing to do with my comment about -v
I was building on your response to the GP. Instead of using -v to see files as they're being deleted, wouldn't it be safer to not use -f, which will not delete important files in the first place?
I see it countless times all over the Internet - everyone is constantly using -rf no matter the situation. The most common horror story in the article is from people using -rf and losing data. The whole point of -f is so normal usage (without -f) doesn't delete important files.
You misunderstand what -f does, however you are right that it is used all over the place by default and it does add more risk than not using -f. However you're wrong that the lack of -f is risk free. Unfortunately even without including the -f flag it is still possible to delete "important" files given that "important" can mean a spectrum of things, both in terms of file system metadata and in terms of personal worth.
In fact `rm`doesn't even try to understand what an important file is (and why should it?). All the -f flag does is ignore prompts and non-existent files:
$ man rm
...
-f, --force
ignore nonexistent files and arguments, never prompt
...
I think it also overrides skipping the removal of files that don't have a write bit set (despite that not being explicit in the man page) but even if that's also true, you're going to have a crap load of important files that are still writable. Thus it's still quite possible to trash your host using `sudo rm -r /`
You might then say "so you shouldn't use sudo either" and that's true. But going back to my earlier point about "what is an important file?", you're still going to have lots of "important" files on your system that aren't root owned. Maybe development files. Maybe personal files. Thus it is trivially easy to remove wanted files with a careless `rm -r ~/$UNSET` (a bug in Steam's Linux installer once did this in fact).
So the benefit of -v isn't mitigated by the non-inclusion of -f. Not even slightly.
+ a command that doesn't actually delete but rather moves the file to a trash folder that gets emptied periodically (this is exactly how a desktop environments work)
+ a command that removes the inode but leaves the data intact. This could get pretty messy pretty quickly on SSDs.
+ `rm` behaves as normal but you restore the file using forensic tools / file recovery tools. Similar problem with SSDs as above.
+ `rm` behaves as normal but you run a snapshotting file system such as ZFS (maybe even have a wrapper function around `rm` to take a snapshot before running `/bin/rm`). Any "undo" would be recovering from snapshot.
+ `rm` behaves as normal but you restore from backup. Which isn't really a solution but still worth presenting here given the range of options I've presented.
Personally I went with ZFS where possible.
If recovery is tricky and I'm throwing wildcards at the problem then I'll do an `ls` first just to double check I'm not doing anything stupid.
There are a number of commands named "trash". But I don't want to keep in mind to execute "trash-empty" occasionally, especially when rm'ing the same large file several times (say, ones made by "dd if=/dev/zero ..."). So either a cron job for emptying the trash, or I think more elegantly, have "trash" just empty the trash before trashing the next file (i.e. there is only ever one thing in the trash).
It's something you learn just because of this kind of experience. For me, it was my attention to detail that spotted it immediately, and the customer facepalmed hard. It's one of those things that is easier for a second party to spot. It's why I read everything that's critical out loud before hitting enter, or publishing, or whatever.
In 1989 my then girlfriend now wife convinced me to go into contracting because it was 3x the pay vs working for our mutual employee, a savings and loan (sort of like a bank if you're not in the US) in receivership.*
Our mainframe guy had a buddy who said he knew unix so I interviewed him and rejected the guy because he clearly did not. The company hired the dude anyway and I went on to a fun contract with Shell where every server was named after mollusks which sounded amusing until you had to login to a box with a twenty letter name.
A few weeks after I left my girlfriend calls and to tell me the head of HR had come by and asked if had "done anything" to their computers? What? Like what? No, of course not. I called someone at the S & L to find Mr. New Unix admin had decided to "optimize" the cabling layout of the Sun servers which had in-rack storage. He didn't know that changing the cabling order changed device names, he didn't keep a map of device names to physical disks, which as a Sybase shop using raw disks was a big mistake. You had to know enough about Sybase to take it into maintenance mode and rename devices and if you got it wrong hello database corruption as the dbs spanned multiple disks. For the disks with mounted filesystems he didn't leave breadcrumbs on the root of each respective disk so he could mount it and figure out how to fix fstab. I don't think he even knew what fstab was. After breaking everything he claimed I had somehow dialed in and my "hacking" was to blame.
*I wasn't angry or upset at the S & L, but it was a valuable lesson for me. Hired as a 'C' programmer to write mortgage bond software when the company discovered I had database skills they also had me write a payroll app, which is how I knew everyone's salary and that the previous sysadmin I worked for made $10k/yr more than me. When I asked for a raise to his pay after officially taking over his job (which I had effectively been doing anyway) I got like a $3k raise and the employer claimed it was the best they could do because of their receivership situation.
That was the impetus of looking for another job because my gf was like if you can get more you need to do it. Fiiiiine. When I turned in my notice, my boss' exact words were "Oh, shit" because he knew he'd been underpaying me and there was no replacing a kid making $24k a year who was still coding mortgage analysis software, acting as their DBA and sysadmin. Even if I was the type to exact some kind of revenge (I was not) the fact they had to hire two or three people to replace me was enough. When my boss magically had money for a raise, thanks to my aforementioned payroll app I got to say one of the most satisfying things I've ever said to anyone: "Vic, I make more than you do, now. What's your counter?" Sorry if that's a pointless, rambling sorry. The nightmare was getting accused of hacking some company because of a mishire.
Supporting traders at Enron Capital & Trade was fun because the traders had root to their SPARCstations. In fact the interview was to fix an unbootable SPARCstation with a compressed kernel, something the support team had happen to them when a user needed more disk space. You had to know Solaris well enough to have the installboot syntax memorized and those workstations were so slow to boot you essentially got one shot at it because it was a timed test.
An hour later, he calls back. Hacked again. Restored again.
An hour later, he calls back. He realizes that the hacker is him! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: