Meet the Other Phone. A phone that grows with your child.

Meet the Other Phone.
A phone that grows with your child.

Buy now

Please or to access all these features

Site stuff

Join our Innovation Panel to try new features early and help make Mumsnet better.

See all MNHQ comments on this thread

More about the Technical side of the attacks on Mumsnet

720 replies

JustineMumsnet · 19/08/2015 11:17

Hi all,
There are have been, understandably, a lot of questions about the tech side of the attack on Mumsnet, so here - courtesy of the tech team is some more detail. We obviously do have to be a bit careful with the details because we don't want to give away information that could help other hackers. Whilst it's true that "security through obscurity" isn't real security, we have no wish to make it easier for a future attacker.

We've spent a lot of time since the attacks began, proactively defending against them, minimising the impact of it and protecting against future attacks. With a busy site like Mumsnet there is a lot of information to go through. When we uncover a new snippet of information, perhaps a new suspicious user account, we have to go back to the start and reanalyze, so it can be slow going at times. We are working with our technology partners who have a lot of experience of these kind of attacks and we have used lots of resources available to us.

Some aspects of our technology stack have already been extensively tested by external specialists. Some of our software code is quite old - nearly as old as Mumsnet itself - and things have moved on a lot over that time. However, we have a program of code review whereby all new code is checked by someone other than the person who created it. It's not perfect and everyone makes mistakes, but we take the quality of our code very seriously.

The Denial Of Service (DOS) attack against Mumsnet was a heavy, sustained attack which initially overwhelmed our ability to respond to legitimate requests. Mumsnet might typically get something like 50-100 requests per second. During the attack we were getting around 17,000 requests per second. Each request carried more data than is normal as well.

The hacking attack on our website was separate from the DOS, though we believe perpetrated by the same person or people. We follow many of the industry's best practices, such as using HTTPS for our login pages, keeping our database separate from our cluster of web servers and not accessible from the internet, and so on. We don't necessarily use the same standards of security as say your online banking service might use, for example requiring multiple passwords or using two factor authentication. We try to balance security against usability and the sensitivity of the information we hold. After all, as pointed out by one of you in an earlier thread, the majority of information we have about a user is what that user publishes in Talk, which is there for all to see.

As has been mentioned several times, we keep our passwords encrypted and we use the recommended algorithms for this, with high "strength" settings. This means that if someone somehow obtained the password data from our database they wouldn't be able to make any use of them - they wouldn't work on our site or on any other site even if the user used the same password on that other site. This remains the case even for MNHQ staff; they cannot un-encrypt the passwords either.

We are now pretty confident it was a phishing attack. Phishing, where a hacker gets a user to enter their username and password into a form from which they can capture that information, fits all the data we have. The hacker doesn't need to decrypt anything, because they capture the password in the browser as it is entered (either by typing it, or if it was automatically remembered by the user's browser or password manager). The list of passwords that has been published includes some that users have identified as being ones that they've mistyped. Our database wouldn't have mistyped ones, only accurate ones, whereas those collected by recording what a user submits would and does contain errors.

It's not obvious how it has been conducted though. We have been able to create a proof of concept which shows that it could work, but that relies on some steps that would be difficult or virtually impossible for a hacker. Phishing attacks sometimes use social engineering to "trick" people into using the fake website rather than the real one, but again, for various reasons, we can rule some of these out. Other phishing attacks are more technical and use other means to get people to visit the fake page. One such example is Cross Site Scripting (XSS). XSS is ranked number three on Open Web Application Security Project top ten list of web site security problems. If the hacker can get the website to put his own code on pages which are to be viewed by other users, s/he can modify the page to either redirect the login process to their own site, to a page which looks just like our login page but is actually recording the details and sending them to the hacker. Also possible, but even less likely, is modifying our login page to submit the details to the hacker as well as to us. If the hacker had gained access to our Content Management System he could have done the former, though not the later. However, we record all changes that are made and there are no suspicious ones.

It's impossible for us to know how many users' passwords have been collected. It's a reasonable assumption, and our working one, that the passwords of everybody that has logged since 6th August 2015, and possibly some time before that, have been collected.

In light of the attacks, we've bolstered some aspects of our security, particularly around our administrative functions. We have further changes planned and will be working on these in the coming days.

Forcing everyone to reset their password, as we have done, would render the list useless provided that users don't choose the same new password and they've not used the same username and password elsewhere.

Some users have questioned why certain other changes aren't being made already, such as a move to enforcing stricter passwords, which makes sense. However, given how crucial the part of our system that deals with passwords is, we have to be really cautious when making changes to it so we don't want to rush and end up creating bigger holes but we will certainly take steps to encourage users to strengthen passwords as soon as practicable.

Any questions do post here - we'll answer as transparently as we can - bearing in mind the caveat about helping future hackers mentioned earlier.

OP posts:
Thread gallery
6
Garrick · 28/08/2015 16:17

Too much MNHQ-knocking going on here. Yes, I am a data bore and, yes, I do care about web security - though my own hacking skillz are so out of date now, I just fiddle around with my phone for fun and I'm not even good at that any more.

But many, many sites are a whole lot worse than MN. Many of those interact financially with their users and/or hold more personal data on them. Having read all the 8chan threads last night (taking frequent mental health breaks), posters did actually say MN was harder to crack than they'd expected. I got the impression they lapsed into long trolling episodes to entertain themselves & keep up the motivation to find a way in. It looks like their exploit would've proved pointless if they hadn't found a couple of weak logins. That isn't MNHQ's fault: the same happened to the White House! Each user needs to be security-aware, as well as the site owners.

In any event, it's great that MN has now got a consultancy on the case :)

Garrick · 28/08/2015 16:19

Forgot to add - Having read the 8chan threads, I ? HoneyBadger Grin

LurkingHusband · 28/08/2015 16:40

Passwords - particularly for non trivial applications - are a tad old-hat.

It should be two-factor authentication all the way (like I was using 10 years ago).

Simurgh · 28/08/2015 16:46

There surely has to be some compromise between ease of access and security, though?

2boysnamedR · 28/08/2015 16:57

I don't think any other chat type community board has a two stage log on?

Garrick · 28/08/2015 16:59

I really don't want to have to bugger about with 2-factor just so I can log into MN! People with operator access, though, should.

LurkingHusband · 28/08/2015 17:00

I wasn't necessarily talking about the users. More the admin.

There's an almost religious ceremony that would be needed if the internets DNS infrastructure needed rebuilding. A group of people from the W3C need to bring their encryption keys together at the same time. Not very easy to hack.

Garrick · 28/08/2015 17:01

Making users jump more hurdles to log in wouldn't help, though, would it? Jeffrey could have infected his profile page just the same.

Garrick · 28/08/2015 17:02

xpost, LH :)

YY, I watched one of their meetings! Weird and impressive in a low-key sort of way.

ItsAllGoingToBeFine · 28/08/2015 17:08

Two-factor authentication isn't a hassle at all, especially if it is optional.

Most services using it allow you to specify trusted devices so you only need to do the two factor bit once.

And even when you do the whole shebang its only an extra few seconds to type in the code.

Would be nice if it was an option - that way people who want extra security with extra hassle can, and those who are more relaxed don't have to (although I do think MN should maintain a certain minimum level of security for everyone)

2boysnamedR · 28/08/2015 17:09

But isn't that guessing?

Discussing admin security isn't something I would share with end users. It's not need to know

securitylecturer · 28/08/2015 17:09

People with operator access, though, should.

And it's easy enough to implement at a first approximation: you restrict access to your administrative functions to your physically secure local network, and you make all remote access (including things like home-working moderators, of which I suspect MN has a lot) be via a VPN protected with two factor (and other mod cons, such as a certificate or a group secret). You can do with quickly with off the shelf hardware, it makes your firewalls easier and it shoudn't require major changes to the core applications. At a second approximation, you treat your office network as red as well and have everyone accessing your administrative functions use the VPN, even from across the corridor.

Neither of these is perfect, because the "restrict access to your administrative functions" bit may not be easy, and in any event will be very hard to assure with anything even remotely approximating formality. But it raises the bar for attackers massively.

Retrofitting security is hard, unfortunately.

ItsAllGoingToBeFine · 28/08/2015 17:10

Jeffrey could have infected his profile page just the same.

Yes, but the released usernames and passwords would be useless if everyone on the list had 2-step authentication set up.

securitylecturer · 28/08/2015 17:35

Yes, but the released usernames and passwords would be useless if everyone on the list had 2-step authentication set up.

Would be useless for accessing MN, at least. They'd still work for other sites with which the users had used the same credentials (as users often do). Which is a tragedy of the commons thing: by implementing 2FA a site reduces the requirements for password security and confidentiality while authenticating, but by continuing to use passwords as part of the 2FA (in fact, as the other F in their A scheme) they potentially weaken security for other sites.

ItsAllGoingToBeFine · 28/08/2015 17:49

Fair enough. It's a tricky balance between protecting people from themselves, and ease of use.

Although I suspect that there would be a strong correlation between those who choose to use 2-factor authentication and not using the same passwords between sites Wink

securitylecturer · 28/08/2015 17:56

Although I suspect that there would be a strong correlation between those who choose to use 2-factor authentication and not using the same passwords between sites

That's relying on the fact that users of 2FA aren't homogenous with the population at large. In large-scale deployments that assumption breaks down.

And of course, the sort of users who voluntarily use 2FA are the sort of users who use password managers to deal with unique passwords and check certificates, which means they are much less susceptible to the attacks that 2FA mitigates anyway. Be a good exam question: "It has been said that Using 2FA heavily suggests you don't need to use 2FA; the people who most need 2FA are the people who don't use it. Discuss. Write on only one side of the paper at once."

ItsAllGoingToBeFine · 28/08/2015 18:03
Grin
Garrick · 28/08/2015 18:18

Write on only one side of the paper at once Grin

HexBramble · 28/08/2015 18:57

I'm still getting
"connection failed. Unable to connect with Mumsnet." On my iPhone App.
I read somewhere up ^ there of an 'old App and new App'. Should I delete the App and re load?

Thanks.

Simurgh · 28/08/2015 19:14

The old App has been 'turned off' HexBramble because of potential security concerns - 'unable to connect with Mumsnet' is the message I'm getting on my ipad. The new App is still some weeks away. Try the mobile site?

JustineMumsnet · 28/08/2015 22:31

@00100001

"Hiya, sorry not quite sure what you mean by happened before? Can you explain a bit further?"

We believe the hacker has used a password from the old hack to gain access to another system (external to Mumsnet) on which we store client information

I'm talking about the time MN was affected by Heartbleed. So, whilst not the same as this phishing hack. Vulnerabilities were exposed. Data lost. Admin at the time had no idea about what was taken etc. All serious stuff.

So, my concern is/was that apparently nothing seems to have been learned Confused Admin passwords were still ridiculously weak...ridiculously (see this example : [email protected]:LisaMumsnet)

The MN website still had vulnerabilities in it, allowing the phishing to happen. Despite being victim to previous attacks. Why was the code not robust enough?

I can see how you'd conflate the two events binary, but there really wasn't any similarity. The weakness exploited during Heartbleed was not a Mumsnet site weakness - it was a flaw in the code of the well-regarded and widely used encryption software - OpenSSL - that many, many sites used. No site could know if they'd fallen victim of the Heartbleed bug - there was simply no way of telling - unless, as in our case, hackers took the trouble to tell you and prove it. That hack didn't happen because of weak admin passwords (and neither did this one).

That's not to say, of course, that we are blameless with regard to this current attack - our code clearly had vulnerabilities. The only thing I can say (and I realise this isn't really any kind of justification) is that according to the experts I've spoken to, very many sites would be equally as vulnerable to a similar type of concerted offensive.

On top of that you're quite right in saying we can (and have) improved our
internal security with regard to admin passwords and some other internal procedures.

But I don't believe there are parallels between the heartbleed exploit and this hack or that the heartbleed attack led in any way to this one.

OP posts:
JustineMumsnet · 28/08/2015 22:46

@ItsAllGoingToBeFine

Oh I see. Well heartbleed was internet-wide vulnerability, so not sure what lessons we could have learned that might have helped us with this situation?

From here: www.mumsnet.com/features/mumsnet-and-heartbleed-as-it-happened

The heartbleed bug was disclosed and fix made available on 7th April. You patched Heartbleed hole on 9th April. You assumed that this hole had not been exploited prior to this date and, as I recall, told users their data was safe. This was shown to be incorrect on the 11th April, users were informed and passwords reset on the 12th April. One might suggest that fix could have been applied faster, and that passwords should have been reset immediately after the fix had been applied.

In the current hack some fairly unskilled hackers attacked using an XSS vulnerability - I think one of the Tech's said in an earlier thread that this was listed as.third.most common vulnerability, it has been known about for decades, yet it would appear.MN did not proof the site in any way against this.

I think a big issue is that MNHQ is using custom software with a small tech team -this makes it very hard for them to keep abreast of and counter against the latest threats in a timely fashion.

There are also issues, shown during Heartbleed and during the current attack with communication with users, and there appears to be no.system or.procedure in place to deal with the prevention of attacks, during an attack, and after an attack.

No I'm pretty sure we didn't assume the heartbleed hole hadn't been exploited and didn't tell users their data was safe because we couldn't know whether the hole had been exploited - no one could. We only knew once the hacker "kindly" told us by impersonating me on a thread and sending an "All your base" message. We also were widely praised for how we dealt with the breach and communicated with users - example here

But you make some valid points about our code and general set up and as a consequence we are investing in external mitigation/security services because as you say, without dedicated security personnel, it's tricky to stay on top of every latest hack/exploit.

OP posts:
JustineMumsnet · 28/08/2015 22:51

@2boysnamedR

I feel bad for MN, I feel extremely bad for Justine and those attacked. MN isn't a big and obvious target for a hit. Not everyone needs to know the in and outs of finite tech details and even the biggest make mistakes. So personally I like to know my data is safe, but I'm not shocked it could happen. Vulnerabilities are discovered daily. Big companies can jump on them and patch ad hoc. I doubt MN can. That's the nature of IT

I think if you work in tech it's very interesting, not at "look I know better". I had security training a while back. I was a bit surprised I learnt a lot of new things, I have then also learned a lot from this.

IT doesn't stand still. Hackers lead this field and security follow.

Pointless post but I value MN. So thanks Justine. Your site means a lot to me. Mistakes happen. Bad people do shit things. You didn't deserve such a crap thing.

Thanks - that's much appreciated, 2boys and you're welcome. Thanks for your (and everyone's) contributions.

OP posts:
JustineMumsnet · 28/08/2015 22:55

@Simurgh

If the reference was to Heartbleed, I'm sure Binary will respond to that, Justine. My own view, however, is that any impact - personal to your system or not - should act as a 'wake-up' moment for any organisation. That refers to anything and not just IT but given that IT is critical to your functioning, I think that that should take a very high priority on the organisation's list.

This is an opportunity for MN to really assess where it stands and become a leader in its field. (That may necessitate some changes to the financial plans but so be it; others will have to catch up at some point.)

You said 'so not sure what lessons we could have learned that might have helped us with this situation?' and I can't advise you on that. It's still fairly clear that there were some, though. Jeffrey's fanclub were making some pretty disparaging remarks about the robustness of MN systems; and whether that is or is not true, it still speaks to 'public' perceptions I think. You could do with altering those.

Completely agree that there are lots of lessons to be learned from this latest series of attacks Simurgh. We've already put in place quite a few new protocols and they'll be more to follow.

OP posts:
Simurgh · 28/08/2015 22:58

Thanks Justine - good to hear that. And well done for coming on here at this time of night and answering questions. Precious few CEOs would take the time to do that I think. Smile