Rant: Customers who know best then decide you were right
- From: Michael Greb <michael@xxxxxxxxxxxx>
- Date: Wed, 01 Mar 2006 16:37:49 -0600
So, I've had my current position for just over a year now. It's a small
company, just the boss and I. We do those virtual server thingies for
web-hosting/email/whatever the customer wants a server for. Since it is
still just the two of us we both do a bit of everything though I handle
most all of the customer service related stuffs. Support tickets,
billing enquiries, etc.
A month or so ago we received a several abuse reports indicating SSH
brute force attacks coming from one of our IPs. I verified that the
traffic did indeed exist and opened an abuse ticket with a customer. He
failed to respond to the ticket and thus his server was shutdown when
the deadline in the ticket was reached. Upon discovering that his
server wasn't running, he logged in to the member's site which is when I
assume he saw the open ticket.
He updated the ticket stating that this was a production server and we
have no right to shut it down blah, blah, blah. He wanted to know which
IPs his system was attacking what IP the attacks supposedly came from
and "I doubt they are originating from my machine.. they could be
IP-Spoofed."
After booting his server back up (they are initially just shutdown, not
suspended as a manner to get the customers attention since the email
didn't work) he found the problem! He had SSH brute force attacks
targeting him for at least the last three days straight! Surely this is
how his server was compromised. I told him unless he had a common
username with the password set exactly to the username it was not very
likely, unless someone was specifically targeting his system, again not
very likely.
He comes back stating of course, that was the problem, I had a username
jon with password jon followed by four digits. I tell him again that
though that password was weak, it almost certainly wasn't the entrance
vector and that he should look at the web based applications he is
running as in the past that is generally the case. He continues to
claim that no, this is how they got in and the problem was fixed. I
reiterate that I don't believe his machine is safe and that should we
receive further complaints about traffic from his machine, we will not
be able to wait as long as the first time before shutting down his
server.
I watched his traffic volume and packet headers religiously for the next
24 hours or so and then did so less frequently. A few days later, sure
enough, more reports arrive. This time, both SSH brute force attacks
and spam. I block outbound port 25 traffic and open a new abuse ticket
for the spam and update the existing ticket. I give him four hours to
fix the problem and respond to the ticket. Four hours come, I shutdown
his server and this time, disable it.
He responds to the spam ticket, now mind you, there was a full copy of
one of the spams in the initial ticket with the recipient's address
removed. He wants to know if there is a way to know if the message came
from localhost or not as only message from 127.0.0.1 are accepted from
relaying. Apparently his time is too valuable to be wasted looking at
the headers that were provided to him.
So I tell him that yes the messages came from localhost, and he needed
to backup his data and reinstall from scratch. He replies asking if
this is really necessary, "Isn't there any other way." I tell him no,
not at this point, he had his chance with the previous ticket when I
told him the problem was not resolved and he wasn't interested in
listening to me. He updates asking couldn't we just block outbound port
25 and 22 access until he fixes the problem and asks what I would
normally check in this situation then repeats that he thinks just
blocking port 25 and port 22 out will be fine until /WE/ can figure out
what happened.
I reiterate that at this point this is his only option. He asks how he
can backup his data without networking so I point him to the
documentation on our website for getting console access via SSH.
At this point we are at several days ago. He asks how he is supposed to
use tar when he can't write to a file (since the disk is mounted RO). I
tell him how to remount his filesystem read/write. About the time that
I'm updating the ticket he shows up in the IRC support channel.
The basic gist of the conversation is "but I did listen to you, you said
look at webapps so I disabled them I also changed all the passwords to
stronger ones because they were weak and that's how they got in". Well,
remember that piece of spam?
Received: from www-data by [redacted to protect the guilty] with local
So 1) he didn't disable the webapps, 2) I was right. Now he is begging
for help finding out what happened so he doesn't have to reinstall. In
addition "I've never had bad customer service when dealing with $boss"
followed by something about me being rude. I told him he was welcome to
speak with $boss but I would not be re-enabling his server until he had
reinstalled.
I went away at this point, outside to smoke a cigarette. I am about to
partake in large quantities of beer followed by a pizza chaser. Then
perhaps I can get some coding done this evening.
I did notice that he did find the culprit in his HTTP logs with other
people in the channel's prodding, every cracker's favorite log analysis
software, awstats.
Michael
.
- Follow-Ups:
- Re: Rant: Customers who know best then decide you were right
- From: Dennis Davis
- Re: Rant: Customers who know best then decide you were right
- Prev by Date: Re: Can I have some fun now, please?
- Next by Date: Re: Can I have some fun now, please?
- Previous by thread: Ho hum
- Next by thread: Re: Rant: Customers who know best then decide you were right
- Index(es):
Relevant Pages
|
