Please or to access all these features

Site stuff

Join our Innovation Panel to try new features early and help make Mumsnet better.

See all MNHQ comments on this thread

More about the Technical side of the attacks on Mumsnet

720 replies

JustineMumsnet · 19/08/2015 11:17

Hi all,
There are have been, understandably, a lot of questions about the tech side of the attack on Mumsnet, so here - courtesy of the tech team is some more detail. We obviously do have to be a bit careful with the details because we don't want to give away information that could help other hackers. Whilst it's true that "security through obscurity" isn't real security, we have no wish to make it easier for a future attacker.

We've spent a lot of time since the attacks began, proactively defending against them, minimising the impact of it and protecting against future attacks. With a busy site like Mumsnet there is a lot of information to go through. When we uncover a new snippet of information, perhaps a new suspicious user account, we have to go back to the start and reanalyze, so it can be slow going at times. We are working with our technology partners who have a lot of experience of these kind of attacks and we have used lots of resources available to us.

Some aspects of our technology stack have already been extensively tested by external specialists. Some of our software code is quite old - nearly as old as Mumsnet itself - and things have moved on a lot over that time. However, we have a program of code review whereby all new code is checked by someone other than the person who created it. It's not perfect and everyone makes mistakes, but we take the quality of our code very seriously.

The Denial Of Service (DOS) attack against Mumsnet was a heavy, sustained attack which initially overwhelmed our ability to respond to legitimate requests. Mumsnet might typically get something like 50-100 requests per second. During the attack we were getting around 17,000 requests per second. Each request carried more data than is normal as well.

The hacking attack on our website was separate from the DOS, though we believe perpetrated by the same person or people. We follow many of the industry's best practices, such as using HTTPS for our login pages, keeping our database separate from our cluster of web servers and not accessible from the internet, and so on. We don't necessarily use the same standards of security as say your online banking service might use, for example requiring multiple passwords or using two factor authentication. We try to balance security against usability and the sensitivity of the information we hold. After all, as pointed out by one of you in an earlier thread, the majority of information we have about a user is what that user publishes in Talk, which is there for all to see.

As has been mentioned several times, we keep our passwords encrypted and we use the recommended algorithms for this, with high "strength" settings. This means that if someone somehow obtained the password data from our database they wouldn't be able to make any use of them - they wouldn't work on our site or on any other site even if the user used the same password on that other site. This remains the case even for MNHQ staff; they cannot un-encrypt the passwords either.

We are now pretty confident it was a phishing attack. Phishing, where a hacker gets a user to enter their username and password into a form from which they can capture that information, fits all the data we have. The hacker doesn't need to decrypt anything, because they capture the password in the browser as it is entered (either by typing it, or if it was automatically remembered by the user's browser or password manager). The list of passwords that has been published includes some that users have identified as being ones that they've mistyped. Our database wouldn't have mistyped ones, only accurate ones, whereas those collected by recording what a user submits would and does contain errors.

It's not obvious how it has been conducted though. We have been able to create a proof of concept which shows that it could work, but that relies on some steps that would be difficult or virtually impossible for a hacker. Phishing attacks sometimes use social engineering to "trick" people into using the fake website rather than the real one, but again, for various reasons, we can rule some of these out. Other phishing attacks are more technical and use other means to get people to visit the fake page. One such example is Cross Site Scripting (XSS). XSS is ranked number three on Open Web Application Security Project top ten list of web site security problems. If the hacker can get the website to put his own code on pages which are to be viewed by other users, s/he can modify the page to either redirect the login process to their own site, to a page which looks just like our login page but is actually recording the details and sending them to the hacker. Also possible, but even less likely, is modifying our login page to submit the details to the hacker as well as to us. If the hacker had gained access to our Content Management System he could have done the former, though not the later. However, we record all changes that are made and there are no suspicious ones.

It's impossible for us to know how many users' passwords have been collected. It's a reasonable assumption, and our working one, that the passwords of everybody that has logged since 6th August 2015, and possibly some time before that, have been collected.

In light of the attacks, we've bolstered some aspects of our security, particularly around our administrative functions. We have further changes planned and will be working on these in the coming days.

Forcing everyone to reset their password, as we have done, would render the list useless provided that users don't choose the same new password and they've not used the same username and password elsewhere.

Some users have questioned why certain other changes aren't being made already, such as a move to enforcing stricter passwords, which makes sense. However, given how crucial the part of our system that deals with passwords is, we have to be really cautious when making changes to it so we don't want to rush and end up creating bigger holes but we will certainly take steps to encourage users to strengthen passwords as soon as practicable.

Any questions do post here - we'll answer as transparently as we can - bearing in mind the caveat about helping future hackers mentioned earlier.

OP posts:
New posts on this thread. Refresh page
Thread gallery
6
BertieBotts · 02/09/2015 20:04

Absolutely you can look on the FAQ thread which is updated with anything major.

SandyMumsnet · 02/09/2015 09:57

Morning everyone,
We wanted to let you know that we have a webchat with Graham Cluley, an award-winning computer security blogger, at 1pm today.

You may want to hop onto this thread and ask him a question.

absolutelyloveit · 31/08/2015 15:20

MNHQ is there any way to be alerted if MNHQ post? I'd like to know if there is a new post about the hacker/site security etc, but don't really need to be alerted to other posters' musings/technical speculation etc!

Garrick · 31/08/2015 14:13

:)

SomethingOfTheNight · 31/08/2015 08:17

Thank you Garrick.

Garrick · 31/08/2015 02:19

Strict answer - it is never safe and you should trash all your cookies after any logged-in session anywhere.

Normal answer - yes, it makes hardly any difference. Always be more paranoid if connected to a public network, like an airport or coffee shop. Your mobile network ought to be about as safe as using your home wifi.

SomethingOfTheNight · 31/08/2015 00:23

In light of hackergate, would those of you in the know please advise whether, when logging in to the mobile site it's safe or not to tick the 'keep me logged in option '

Thank you

HexBramble · 29/08/2015 20:19

Thanks Simurgh.
Will delete App and be patient.
Thanks to all, especially MNHQ.

Garrick · 29/08/2015 16:54

Oh, OK. I thought you'd misinterpreted something :)

Simurgh · 29/08/2015 16:09

Goodness Garrick. Just a dropping in of an idle piece of information. No more than that. Smile

Garrick · 29/08/2015 15:16

How did you find out GCHQ had intercepted your emails?

Some quite tedious checking, pinging and googling. I got an expert pal to check my work before politely contacting someone to ask them to desist. My request probably made no difference, tbh, but it did stop a few days later. I was quite proud of having tracked it, but found it hard work. I learned my character isn't suited to doing network stuff for a living!

I respectfully submit that [the NSA] doesn't sound like a One Man and His Dog outfit - I don't understand where this came from?

DontHaveAUsername · 29/08/2015 14:54

I have to agree with Garrick, I use a password manager to store all my passwords, so every time I register on a new site I can generate a very long complex password that no one would guess, and I never even need to remember it. And if that password ever is compromised, I only need to change that one password, as I'm not using it for everything. You only need to remember one password, the one that unlocks your database and everyone can do that.

Simurgh · 29/08/2015 08:38

'I have a suspicion that has been a user error somewhere along the way'

Grin Good luck.

Simurgh · 29/08/2015 08:37

'twas ever thus, Garrick.

Let's not forget that in 2014, the NSA recycled - actually dealt with (in recorded fashion) in one year alone - 108,000 pounds of ceiling tiles. I respectfully submit that that doesn't sound like a One Man and His Dog outfit living in the shed at the bottom of the garden.

Mistakes happen but if Mumsnet and its users learn from this one, I'll be a little happier.

TheHoneyBadger · 29/08/2015 08:01

lots most of this is way over my head.

do remember though that it is not just usernames and passwords that have been taken as many people sign in with their email address - i'm not trying to be pedantic but to ensure people don't forget that just because their email wasn't published doesn't mean it isn't out there with people trying to hack it and you should consider what you use that particular email account for and probably change and strengthen your password even if it isn't the same password as you used here.

the other thing that was interesting was to read about what happened in april - i was traveling and doubt i was on much or at all during heartbleed but i certainly never had to change my password then or since until these recent events so possibly the 'force log out/password change' (whatever that entails tech wise) has never been 100% effective. i know this time there were still people saying they hadn't been logged out or forced to change password and had to do it voluntarily instead.

thanks justine for more answers. very much hope it's over with now.

Sansoora · 29/08/2015 08:01

Garrick you and Sadwidow write about the most exciting of adventures.

How did you find out GCHQ had intercepted your emails?

FuckitAndStartAgain · 29/08/2015 07:56

Fed up of the point scoring here, which is a shame because I find the whole Web stuff fascinating and have learnt a lot of the last few days.

MNet is a huge community, a business hence Justine as a CEO, but foremost a community, hence the updating which is superb in comparison to other websites that have been compromised.

  1. I don't think explaining all the ins and outs of security as pertaining to MNet here is appropriate, the hackers read this, others learn etc etc.
  1. If you don't like it, leave it. It is not Law that you have to give information to MNet, binary! If you are not sure about it right now, leave and considering returning at some point in the future.
  1. I reckon there are some tired and stressed people, if posts are not totally clear or information changes can't we just accept that's what happens? It is fast moving and exhausting fielding security issues. Few companies would add to their lists of tasks to communicate with clients and try and maintain services as do MNet.

Right now that is off my chest I am going to check out what is making my last pass account unreliable, I have a suspicion that has been a user error somewhere along the way ????

Garrick · 29/08/2015 01:30

Sorry, by 80% I meant including all the CCTV, other people's phones, hacked wifi points and so on. All our comms are intercepted, but nothing's done with them unless we've been supposedly identified as a security threat. Mine were for a couple of weeks 4 or 5 years ago, probably because I was engaged in some heated arguments about Palestine (not on MN.) All my emails were coming in late and I found out why. I'm sure the heuristics are a lot more sophisticated now.

ChristineDePisan · 29/08/2015 01:23

I'd modify your statement, Garrick, to say that GSHQ have the capability to intercept all your internet activity, but they aren't capturing 80% of everything we do on MN

I think MN have stepped up their game significantly since the initial hacking - well done and thank you Flowers

Garrick · 29/08/2015 01:16

I agree, MN are handling this beautifully.

For those who say the XSS vulnerability shouldn't have been there - on today's interweb, they are almost inevitable. The problem is that dynamic web pages - those that contain lots of different responsive stuff, rather than just a page of writing - load content in from masses of different places, by no means all of them under the owner's control. All this content can & should be double-checked before loading, however there's always a balance to be struck between time to serve and security. Explanation here.

Luckily, most malicious content will do no worse than install some annoying ad toolbar on your browser but that is not guaranteed. It can't be.

What sites can do correctly is implement rock-solid security at their end. This must include everything from changing staff passwords frequently, to updating the back-end code each and every time anything at all changes. The latter is a massive job, seriously. You pretty much have to rely on security bulletins about new vulnerabilites - by and large, those bulletins come after hackers have discovered them, so somebody suffered.

As an internet user - Get a password manager (Google now does a free one, if you don't mind using Chrome all the time) and change your passwords frequently. If you can remember your password, someone can guess it.

And finally - look away now if you're paranoid about privacy! - GCHQ intercepts every single thing you send over the internet or by phone. There's some device recording you more than 80% of the time. The only way to have a truly private conversation would be to meet down an undiscovered pothole! MN just wouldn't be the same that way Wink Though I guess it might be fun.

Simurgh · 28/08/2015 22:58

Thanks Justine - good to hear that. And well done for coming on here at this time of night and answering questions. Precious few CEOs would take the time to do that I think. Smile

JustineMumsnet · 28/08/2015 22:55

@Simurgh

If the reference was to Heartbleed, I'm sure Binary will respond to that, Justine. My own view, however, is that any impact - personal to your system or not - should act as a 'wake-up' moment for any organisation. That refers to anything and not just IT but given that IT is critical to your functioning, I think that that should take a very high priority on the organisation's list.

This is an opportunity for MN to really assess where it stands and become a leader in its field. (That may necessitate some changes to the financial plans but so be it; others will have to catch up at some point.)

You said 'so not sure what lessons we could have learned that might have helped us with this situation?' and I can't advise you on that. It's still fairly clear that there were some, though. Jeffrey's fanclub were making some pretty disparaging remarks about the robustness of MN systems; and whether that is or is not true, it still speaks to 'public' perceptions I think. You could do with altering those.

Completely agree that there are lots of lessons to be learned from this latest series of attacks Simurgh. We've already put in place quite a few new protocols and they'll be more to follow.

OP posts:
JustineMumsnet · 28/08/2015 22:51

@2boysnamedR

I feel bad for MN, I feel extremely bad for Justine and those attacked. MN isn't a big and obvious target for a hit. Not everyone needs to know the in and outs of finite tech details and even the biggest make mistakes. So personally I like to know my data is safe, but I'm not shocked it could happen. Vulnerabilities are discovered daily. Big companies can jump on them and patch ad hoc. I doubt MN can. That's the nature of IT

I think if you work in tech it's very interesting, not at "look I know better". I had security training a while back. I was a bit surprised I learnt a lot of new things, I have then also learned a lot from this.

IT doesn't stand still. Hackers lead this field and security follow.

Pointless post but I value MN. So thanks Justine. Your site means a lot to me. Mistakes happen. Bad people do shit things. You didn't deserve such a crap thing.

Thanks - that's much appreciated, 2boys and you're welcome. Thanks for your (and everyone's) contributions.

OP posts:
JustineMumsnet · 28/08/2015 22:46

@ItsAllGoingToBeFine

Oh I see. Well heartbleed was internet-wide vulnerability, so not sure what lessons we could have learned that might have helped us with this situation?

From here: www.mumsnet.com/features/mumsnet-and-heartbleed-as-it-happened

The heartbleed bug was disclosed and fix made available on 7th April. You patched Heartbleed hole on 9th April. You assumed that this hole had not been exploited prior to this date and, as I recall, told users their data was safe. This was shown to be incorrect on the 11th April, users were informed and passwords reset on the 12th April. One might suggest that fix could have been applied faster, and that passwords should have been reset immediately after the fix had been applied.

In the current hack some fairly unskilled hackers attacked using an XSS vulnerability - I think one of the Tech's said in an earlier thread that this was listed as.third.most common vulnerability, it has been known about for decades, yet it would appear.MN did not proof the site in any way against this.

I think a big issue is that MNHQ is using custom software with a small tech team -this makes it very hard for them to keep abreast of and counter against the latest threats in a timely fashion.

There are also issues, shown during Heartbleed and during the current attack with communication with users, and there appears to be no.system or.procedure in place to deal with the prevention of attacks, during an attack, and after an attack.

No I'm pretty sure we didn't assume the heartbleed hole hadn't been exploited and didn't tell users their data was safe because we couldn't know whether the hole had been exploited - no one could. We only knew once the hacker "kindly" told us by impersonating me on a thread and sending an "All your base" message. We also were widely praised for how we dealt with the breach and communicated with users - example here

But you make some valid points about our code and general set up and as a consequence we are investing in external mitigation/security services because as you say, without dedicated security personnel, it's tricky to stay on top of every latest hack/exploit.

OP posts:
JustineMumsnet · 28/08/2015 22:31

@00100001

"Hiya, sorry not quite sure what you mean by happened before? Can you explain a bit further?"

We believe the hacker has used a password from the old hack to gain access to another system (external to Mumsnet) on which we store client information

I'm talking about the time MN was affected by Heartbleed. So, whilst not the same as this phishing hack. Vulnerabilities were exposed. Data lost. Admin at the time had no idea about what was taken etc. All serious stuff.

So, my concern is/was that apparently nothing seems to have been learned Confused Admin passwords were still ridiculously weak...ridiculously (see this example : [email protected]:LisaMumsnet)

The MN website still had vulnerabilities in it, allowing the phishing to happen. Despite being victim to previous attacks. Why was the code not robust enough?

I can see how you'd conflate the two events binary, but there really wasn't any similarity. The weakness exploited during Heartbleed was not a Mumsnet site weakness - it was a flaw in the code of the well-regarded and widely used encryption software - OpenSSL - that many, many sites used. No site could know if they'd fallen victim of the Heartbleed bug - there was simply no way of telling - unless, as in our case, hackers took the trouble to tell you and prove it. That hack didn't happen because of weak admin passwords (and neither did this one).

That's not to say, of course, that we are blameless with regard to this current attack - our code clearly had vulnerabilities. The only thing I can say (and I realise this isn't really any kind of justification) is that according to the experts I've spoken to, very many sites would be equally as vulnerable to a similar type of concerted offensive.

On top of that you're quite right in saying we can (and have) improved our
internal security with regard to admin passwords and some other internal procedures.

But I don't believe there are parallels between the heartbleed exploit and this hack or that the heartbleed attack led in any way to this one.

OP posts: