The Inquirer-Home

BBC blames extended online and iPlayer downtime on heavy load

Move along please
Thu Jul 24 2014, 08:51
BBC London Logo

THE BBC HAS EXPLAINED the scenario that plunged its online services into a black hole at the weekend, putting it down to technical problems.

While speculation was rife, we had heard that an attack had floored the BBC, but it was actually a victim of its own success.

Richard Cooper, the BBC's controller of digital distribution for BBC future media, explained all in a blog post.

He said that two issues lead to the "serious incident" and that the first saw the BBC's clip and metadata server estate, 68 servers in total, fail after the load on a database "went through the roof".

"At almost the same time we had a second problem. We use a caching layer in front of most of the products on BBC Online, and one of the pools failed. The products managed by that pool include BBC iPlayer and the BBC homepage, and the failure made all of those products inaccessible," he added. "That opened up a major incident at the same time on a second front."

Recovering from the failure was not easy and Cooper said that forensics are still been carried out on the cause. Also tough was restoring the metadata services, and the digital lead said that restoration requires more than the old turn it off and on again. He apologised for the outage, adding that this sort of thing is rare.

"BBC iPlayer is an incredibly popular product, last year alone we had [three] billion requests and instances like this are incredibly rare," he said.

"We will now be completing the forensics to make sure that we've fully understood the root causes, and put in place the measures necessary to minimise the chances of such interruptions in the future. We're sorry for the inconvenience."

Earlier this week the BBC told The INQUIRER that its outage was over, despite appearances.

We had asked the BBC why it was preventing people from watching Eastenders on the toilet and it told us that a problem somewhere meant that something was happening.

"We're aware of an issue which means some people can't access certain parts of BBC Online. We're working hard to fix this as soon as possible," it said in a statement at around lunchtime.

We were promised more information and more information came through. That follow up message came around tea time, and told us that as far as the BBC could tell its time in the wilderness was over.

"BBC iPlayer, BBC iPlayer Radio and other parts of BBC Online that were affected by problems over the weekend are now up and running. Our teams continue to investigate the problem to ensure this doesn't happen again," it said. "We will be publishing more details about the problem in due course on the BBC's internet blog."

This was echoed through the BBC iPlayer Twitter account, which confirmed again, that there was plenty to see there.

We experienced problems with the iPlayer radio app until at least 10.30pm on Monday evening, and asked the BBC about that the day after. It did not respond.

Nor did it respond to our request for comment on suggestions that it's time away from ears and eyeballs was the result of a denial of service attack, a theory mooted by security firm Corero.

"DDoS as a cyber-demonstration technique is becoming more popular as the battle for the attention of the public at large rages on," said Corero CEO Ashley Stephenson.

"In this case, large streaming content providers, like BBC iplayer present themselves as an interesting target, in that video is notoriously susceptible to very small changes in bandwidth, latency or jitter and is therefore very easy to disrupt if there is no proactive protection mechanism in place." µ


Share this:

blog comments powered by Disqus
Subscribe to INQ newsletters

Sign up for INQbot – a weekly roundup of the best from the INQ

INQ Poll

Happy new year!

What tech are you most looking forward to in 2015