Meet the Other Phone. Child-safe in minutes.

Meet the Other Phone.
Child-safe in minutes.

Buy now

Please or to access all these features

More about the Technical side of the attacks on Mumsnet

77 replies

JustineMumsnet · 19/08/2015 11:17

Hi all,
There are have been, understandably, a lot of questions about the tech side of the attack on Mumsnet, so here - courtesy of the tech team is some more detail. We obviously do have to be a bit careful with the details because we don't want to give away information that could help other hackers. Whilst it's true that "security through obscurity" isn't real security, we have no wish to make it easier for a future attacker.

We've spent a lot of time since the attacks began, proactively defending against them, minimising the impact of it and protecting against future attacks. With a busy site like Mumsnet there is a lot of information to go through. When we uncover a new snippet of information, perhaps a new suspicious user account, we have to go back to the start and reanalyze, so it can be slow going at times. We are working with our technology partners who have a lot of experience of these kind of attacks and we have used lots of resources available to us.

Some aspects of our technology stack have already been extensively tested by external specialists. Some of our software code is quite old - nearly as old as Mumsnet itself - and things have moved on a lot over that time. However, we have a program of code review whereby all new code is checked by someone other than the person who created it. It's not perfect and everyone makes mistakes, but we take the quality of our code very seriously.

The Denial Of Service (DOS) attack against Mumsnet was a heavy, sustained attack which initially overwhelmed our ability to respond to legitimate requests. Mumsnet might typically get something like 50-100 requests per second. During the attack we were getting around 17,000 requests per second. Each request carried more data than is normal as well.

The hacking attack on our website was separate from the DOS, though we believe perpetrated by the same person or people. We follow many of the industry's best practices, such as using HTTPS for our login pages, keeping our database separate from our cluster of web servers and not accessible from the internet, and so on. We don't necessarily use the same standards of security as say your online banking service might use, for example requiring multiple passwords or using two factor authentication. We try to balance security against usability and the sensitivity of the information we hold. After all, as pointed out by one of you in an earlier thread, the majority of information we have about a user is what that user publishes in Talk, which is there for all to see.

As has been mentioned several times, we keep our passwords encrypted and we use the recommended algorithms for this, with high "strength" settings. This means that if someone somehow obtained the password data from our database they wouldn't be able to make any use of them - they wouldn't work on our site or on any other site even if the user used the same password on that other site. This remains the case even for MNHQ staff; they cannot un-encrypt the passwords either.

We are now pretty confident it was a phishing attack. Phishing, where a hacker gets a user to enter their username and password into a form from which they can capture that information, fits all the data we have. The hacker doesn't need to decrypt anything, because they capture the password in the browser as it is entered (either by typing it, or if it was automatically remembered by the user's browser or password manager). The list of passwords that has been published includes some that users have identified as being ones that they've mistyped. Our database wouldn't have mistyped ones, only accurate ones, whereas those collected by recording what a user submits would and does contain errors.

It's not obvious how it has been conducted though. We have been able to create a proof of concept which shows that it could work, but that relies on some steps that would be difficult or virtually impossible for a hacker. Phishing attacks sometimes use social engineering to "trick" people into using the fake website rather than the real one, but again, for various reasons, we can rule some of these out. Other phishing attacks are more technical and use other means to get people to visit the fake page. One such example is Cross Site Scripting (XSS). XSS is ranked number three on Open Web Application Security Project top ten list of web site security problems. If the hacker can get the website to put his own code on pages which are to be viewed by other users, s/he can modify the page to either redirect the login process to their own site, to a page which looks just like our login page but is actually recording the details and sending them to the hacker. Also possible, but even less likely, is modifying our login page to submit the details to the hacker as well as to us. If the hacker had gained access to our Content Management System he could have done the former, though not the later. However, we record all changes that are made and there are no suspicious ones.

It's impossible for us to know how many users' passwords have been collected. It's a reasonable assumption, and our working one, that the passwords of everybody that has logged since 6th August 2015, and possibly some time before that, have been collected.

In light of the attacks, we've bolstered some aspects of our security, particularly around our administrative functions. We have further changes planned and will be working on these in the coming days.

Forcing everyone to reset their password, as we have done, would render the list useless provided that users don't choose the same new password and they've not used the same username and password elsewhere.

Some users have questioned why certain other changes aren't being made already, such as a move to enforcing stricter passwords, which makes sense. However, given how crucial the part of our system that deals with passwords is, we have to be really cautious when making changes to it so we don't want to rush and end up creating bigger holes but we will certainly take steps to encourage users to strengthen passwords as soon as practicable.

Any questions do post here - we'll answer as transparently as we can - bearing in mind the caveat about helping future hackers mentioned earlier.

JustineMumsnet · 25/08/2015 13:59

Hello again,
Here's the latest update on what's occurred.

Yesterday evening we were hit by another denial of service attack which meant we offline until this morning; as soon as we got back up we were attacked again. This attack was double the size of the previous one and was distributed across many servers but we have no reason to believe that any security breaches occurred, the intention was to take the site offline rather than to hack into it.

We are contracting external DDoS protection providers to help deal with future issues. Many thanks for your understanding - apologies again for the interruption to normal service.

The police are continuing their investigation.

[even more gin]

JustineMumsnet · 25/08/2015 14:13

@Boofhead

I've had two emails from MN Contact Us. One titled 'Mumsnet Calling', and the other 'READ THIS..'

Have these come from MNHQ? (I was one of the 3000).

The first one has someone else's user name on it, so it seems suspicious.

Thanks

Hi Boofhead - I've flagged this to the team to check - I'm not entirely sure what those mails were titled - but we'll let you know soonest if there's anything to be concerned about.

KateSMumsnet · 25/08/2015 16:39

@WandaFuca

On one of my attempts to access the site, I got an error message I hadn't seen before:

"Error Code 20
The proxy failed to connect to the web server, due to TCP connection timeout."

Then my IP address, Proxy IP address, Origin Server IP.

Anyone know what that was all about?

We think this was to do with us getting the site back online, but if you see it again please shout!

TheOnlyOliviaMumsnet · 25/08/2015 20:02

Shopafrolic - you should have x 2 mails
Apols for any additional anxiety caused.
Thanks

JustineMumsnet · 25/08/2015 22:19

We do have 2-step verification in place for staff mails. ta for thought.

KateSMumsnet · 27/08/2015 10:55

We're here! So sorry folks, know you've got a lot of questions that need addressing. We're working our way through the thread, trying to get to the bottom of everything - please bear with us.

SarahMumsnet · 27/08/2015 11:58

@00100001

Why is no-one fro HQ commenting on the questions
  1. were you or were you not made aware of this breach in the middle if July

  2. why, after the previous hack, was security not tightened?

  3. why were admin passwords so very simple?

Hi 00100001

Sorry for the slow response; we're just getting back up to speed. In answer to your questions:

  1. we weren't aware of the breach in mid-July; it was only after the DDoS attack and hacking in August that we found out about it.

  2. had we known, we would of course have tightened security at the time, and informed all Mumsnet users about it

  3. You’re quite right: admin passwords should have been better. Conversations have been had, and internal security of every kind has been tightened up. While it’s worth bearing in mind that the initial hacking wasn’t performed via forcing/guessing passwords, it’s nevertheless been a salutary reminder for everyone to prioritise security when it comes to passwords, both on MN and elsewhere.

JaneMumsnet · 27/08/2015 11:59

Hello,

Really pleased to confirm that next week, Graham Cluley will be joining us for a webchat on cybersecurity.

We'll post a link to the webchat thread once live, but for now, please put next Wednesday 2 September, 1pm-2pm in your diary.

Thanks

MNHQ

SarahMumsnet · 27/08/2015 12:06

@00100001

4) when MNHQ said We believe the hacker has used a password from the old hack to gain access to another system (external to Mumsnet) on which we store client information does that mean you didn't change the password on other systems?

Apologies for not updating you on this. We've since found out that the hacker did NOT gain access to another system. We realised that the list of client emails was in fact a very old one, from several years ago - i.e. the point when we moved all our client information on to an external system. After a dig around we found that the old list was in fact stored on our own system, and the hacker had picked it up from there. Obviously this is good news, in that the hacker didn't get access to anywhere else - but nevertheless, we changed all passwords across all systems to make sure that security wasn't compromised anywhere. HTH

SarahMumsnet · 27/08/2015 12:25

@Simurgh

'Conversations have been had'.....

I'll bet they have! Grin

Thanks for that, Sarah. I assume that you'll shortly be notifying everyone of the ground rules for the webchat?

Several conversations, Simurgh ...

I'll defer to JaneMumsnet on the webchat rules, as she's set it up; it was only finalised this morning, but we'll update you with more information asap.

SarahMumsnet · 27/08/2015 12:42

One more thing: people have been asking on Twitter and elsewhere about the security of the Mumsnet app, because it uses http, rather than https.

We're going to take the app offline, as we can't be sure it's secure. We'll be launching a new one using https in a few weeks, and in the meantime, we'd encourage everyone to use our mobile site instead. Sorry to all the app users for the inconvenience.

SarahMumsnet · 27/08/2015 13:09

@twirlypoo

Can I make a polite suggestion that you advertise everywhere that the app is being taken down or people are going to panic and think you have been hacked again when they can't get online (well, it's what I would have assumed if I hadn't seen this anyway!)

absolutely twirlypoo - in fact I'm just in the middle of starting an OP in site stuff, which we'll sticky in active/chat/aibu etc. We'll also put up a message on the front page of the current app, explaining that it's been taken down.

SarahMumsnet · 27/08/2015 13:10

@tigerscameatnight

Did anyone ever answer my question as to how long Mumsnet stores our addresses for in regards to product tests and such

I'll pass this on to the insight team, tigerscameatnight, and get an answer for you.

JaneMumsnet · 27/08/2015 13:54

@Simurgh

'Conversations have been had'.....

I'll bet they have! Grin

Thanks for that, Sarah. I assume that you'll shortly be notifying everyone of the ground rules for the webchat?

Hello Simurgh,

We're ironing out the details now. If there's anything that differs from our usual webchat guidelines, we'll make it clear on the thread.

Thanks
MNHQ

SarahMumsnet · 27/08/2015 14:27

Hey all, just to make sure everyone sees this: Justine has posted over here to say that we're taking down our app, effective immediately. I've pasted what she says below, but better to post app-y questions on the other thread, to keep all responses in one place.

Afternoon all,

In the wake of the recent hacking and DDoS attacks, we've been considering the security of the entry points to Mumsnet. Because the current Talk app uses http, rather than https we can't guarantee that it is 100% secure, so we've taken it offline.

We've been developing a new ios app using https for a while and we'll be launching it in a few weeks' time; obviously we'll let you know as soon as it's out. We hope to follow it up with an Android app in due course. In the meantime, though, we'd suggest app users move over to our mobile site. Sorry for the inconvenience; hope to see you on the new one very soon.

Thanks,
MNHQ Flowers

SarahMumsnet · 27/08/2015 14:33

@TheHoneyBadger

in light of this, and ongoing attacks and reassurances that prove false (re: they have no email addresses - well of course they do if they phished from the log in page and many people use their email address to log in) i am not happy about the lack of proper response to people asking about names and addresses stored by mn for prize winners, survey takers etc. saying 'they're stored separately and we change passwords frequently' is not enough. i for one would like any such data deleted and am shocked that it has ever been 'held on file' when there is clearly no need for it to be. can we have it deleted? a yes or no is a sufficient answer to that really - though if it is no i'd like to understand why we don't have the right to ask you to delete info on us that we didn't even knowingly agree to being stored.

Hi TheHoneyBadger

We routinely delete the data we collect for Insight, competitions and so on. None of that personal information (addresses etc) is ever stored on site.

We also keep address details in spreadsheets for up to three months after a product test ends. We keep hold of this info for this time period just in case there is a problem with the product, and we need to sort something out. The addresses are deleted after the three months are up.

If you’re part of the Insight Panel, we do store your details - but on a secure site, completely separate from Mumsnet. If you ask to be removed from the Insight Panel, we will delete your details from there.

Obviously, if at any time you want us to remove your information, just drop us a line and we’ll sort it out for you.

KateSMumsnet · 27/08/2015 14:54

@TheHoneyBadger

i DO feel sorry for mumsnet, i do attribute the blame on the perpetrator but i am not impressed that there were hacks made in july, moderator accounts accessed back in july and the attacks reported to mnhq back then by tumblr users without us being warned allowing this situation to go forward and for many of us to have our data stolen and published. when we found the pages on which the discussion from the july attacks had taken place and reported to mnhq they said they had become aware of those pages the same day we'd reported them - yet tumblr users (a group offended by the way the hackers had been representing a particular sub-culture on the boards here as part of their trolling whilst using accounts to report posts and use their profile ((once the mod went to it)) to phish the mods powers and access) reported it to mn with links to the site the hackers had been chatting on back in july.

Just to reiterate what SarahMumsnet said upthread - we were not aware of any breach on the site in July. We don’t think we’re aware of this tumblr activity, are you able to send us a link or point us towards the reports?

KateSMumsnet · 27/08/2015 16:10

Jason is right, we've suspended the app for the time being, Justine has explained fully here www.mumsnet.com/Talk/site_stuff/2456686-Mumsnet-Talk-Apps.

JustineMumsnet · 27/08/2015 21:56

@00100001

Because of this

I'll be honest your posts pissed off some furries so we took the liberty of emailing them a link to this thread." and What we're saying is mumsnet now have a link to this thread,

I don't believe we received any mail with a link to this 8chan thread in July. The first we knew of it was in August when a Mumsnet user - Marchlikeanant - linked to it here.

I'll triple check with the community team in the morning but I'm sure they would have raised it had it come into us.

JustineMumsnet · 27/08/2015 21:57

@Spotsondots

Hi. I've been sent an email saying I requested to reset my password, which I did not. I've already reset my password following the DDOS forced logout. Should I be concerned? Tia.

Hi Spotsondots, thanks - we'll let Tech know and get back to you if there's any cause for concern.

JustineMumsnet · 28/08/2015 11:44

@00100001

errr sans no, I think people are genuinely interested in how this happened, especially since it's happened before and on the surface, it didn't look like any lessons had been learned.

It will die down, but if people want reassurance and answers, why can't they have them? Confused

Hiya, sorry not quite sure what you mean by happened before? Can you explain a bit further?

JustineMumsnet · 28/08/2015 12:44

@Simurgh

I suspect that Binary meant Heartbleed - they'll doubtless correct me if I'm wrong.

Oh I see. Well heartbleed was internet-wide vulnerability, so not sure what lessons we could have learned that might have helped us with this situation?

JustineMumsnet · 28/08/2015 22:31

@00100001

"Hiya, sorry not quite sure what you mean by happened before? Can you explain a bit further?"

We believe the hacker has used a password from the old hack to gain access to another system (external to Mumsnet) on which we store client information

I'm talking about the time MN was affected by Heartbleed. So, whilst not the same as this phishing hack. Vulnerabilities were exposed. Data lost. Admin at the time had no idea about what was taken etc. All serious stuff.

So, my concern is/was that apparently nothing seems to have been learned Confused Admin passwords were still ridiculously weak...ridiculously (see this example : [email protected]:LisaMumsnet)

The MN website still had vulnerabilities in it, allowing the phishing to happen. Despite being victim to previous attacks. Why was the code not robust enough?

I can see how you'd conflate the two events binary, but there really wasn't any similarity. The weakness exploited during Heartbleed was not a Mumsnet site weakness - it was a flaw in the code of the well-regarded and widely used encryption software - OpenSSL - that many, many sites used. No site could know if they'd fallen victim of the Heartbleed bug - there was simply no way of telling - unless, as in our case, hackers took the trouble to tell you and prove it. That hack didn't happen because of weak admin passwords (and neither did this one).

That's not to say, of course, that we are blameless with regard to this current attack - our code clearly had vulnerabilities. The only thing I can say (and I realise this isn't really any kind of justification) is that according to the experts I've spoken to, very many sites would be equally as vulnerable to a similar type of concerted offensive.

On top of that you're quite right in saying we can (and have) improved our
internal security with regard to admin passwords and some other internal procedures.

But I don't believe there are parallels between the heartbleed exploit and this hack or that the heartbleed attack led in any way to this one.

JustineMumsnet · 28/08/2015 22:46

@ItsAllGoingToBeFine

Oh I see. Well heartbleed was internet-wide vulnerability, so not sure what lessons we could have learned that might have helped us with this situation?

From here: www.mumsnet.com/features/mumsnet-and-heartbleed-as-it-happened

The heartbleed bug was disclosed and fix made available on 7th April. You patched Heartbleed hole on 9th April. You assumed that this hole had not been exploited prior to this date and, as I recall, told users their data was safe. This was shown to be incorrect on the 11th April, users were informed and passwords reset on the 12th April. One might suggest that fix could have been applied faster, and that passwords should have been reset immediately after the fix had been applied.

In the current hack some fairly unskilled hackers attacked using an XSS vulnerability - I think one of the Tech's said in an earlier thread that this was listed as.third.most common vulnerability, it has been known about for decades, yet it would appear.MN did not proof the site in any way against this.

I think a big issue is that MNHQ is using custom software with a small tech team -this makes it very hard for them to keep abreast of and counter against the latest threats in a timely fashion.

There are also issues, shown during Heartbleed and during the current attack with communication with users, and there appears to be no.system or.procedure in place to deal with the prevention of attacks, during an attack, and after an attack.

No I'm pretty sure we didn't assume the heartbleed hole hadn't been exploited and didn't tell users their data was safe because we couldn't know whether the hole had been exploited - no one could. We only knew once the hacker "kindly" told us by impersonating me on a thread and sending an "All your base" message. We also were widely praised for how we dealt with the breach and communicated with users - example here

But you make some valid points about our code and general set up and as a consequence we are investing in external mitigation/security services because as you say, without dedicated security personnel, it's tricky to stay on top of every latest hack/exploit.

JustineMumsnet · 28/08/2015 22:51

@2boysnamedR

I feel bad for MN, I feel extremely bad for Justine and those attacked. MN isn't a big and obvious target for a hit. Not everyone needs to know the in and outs of finite tech details and even the biggest make mistakes. So personally I like to know my data is safe, but I'm not shocked it could happen. Vulnerabilities are discovered daily. Big companies can jump on them and patch ad hoc. I doubt MN can. That's the nature of IT

I think if you work in tech it's very interesting, not at "look I know better". I had security training a while back. I was a bit surprised I learnt a lot of new things, I have then also learned a lot from this.

IT doesn't stand still. Hackers lead this field and security follow.

Pointless post but I value MN. So thanks Justine. Your site means a lot to me. Mistakes happen. Bad people do shit things. You didn't deserve such a crap thing.

Thanks - that's much appreciated, 2boys and you're welcome. Thanks for your (and everyone's) contributions.

Watch this thread for updates

Tap "Watch" to get all the latest updates