How Digg.com benefits from LAMP usage

Digg.com makes full use of its LAMP (Linux, Apache, MySQL and PHP) server cluster which successfully aids in the efficient news compilation coupled with speed amid the fierce competition.

The site lets its users vote for their favorite stories that are hosted on other websites. The site boasts of 1.2 million users. The site started off in late 2004 as a single Linux server running Apache 1.3, PHP 4, and MySQL 4.0 using the default MyISAM storage engine. Today the site has 100 servers scattered in multiple data centers that host a total of 30GB of data.

With increasing favoritism among the users, the site started to use a load balancer in the front that sends queries to PHP servers, MySQL slave servers that feed the PHP servers, and a MySQL master server that feeds data to the slaves.

In a bid to preempt the transfer of raw queries against the database, the site has implemented the usage of a software called Memcached. This software was first developed for the livejournal site. Memcache facilitates websites like digg.com that need to serve web pages with constantly changing content.

Memcached stores chunks of data that can be extracted and used to dynamically create a Web page. The normal caching systems would slow down performance for such dynamic and comprehensive websites. The other unique feature of Digg’s setup is the use of “sharding”.

Sharding typically involves breaking a database into smaller parts in order to isolate heavy loads for better performance. According to Tim Ellis, a Digg engineer, “If 90 percent of your data is within a certain range, and you can get that part working really fast, then you can help customers. Then it’s OK if the remaining 10 percent is slower”.

A database can be sharded by table, date or range. It involves portioning out data to different physical machines. MySQL does not natively allow sharding. However it does support partitioned tables, federated tables and clusters.

Although sharding aids in speedy performance, the downside that remains is increased complexity. This result in extra effort and work pressure for the database administrators and developers as they are unable to use common SQL commands such as joining tables.
Digg’s current assemblage includes about 20 database servers, 30 Web servers, and a few search servers running Lucene; the balance operate as backup servers. The heavy transaction servers as well as the backup units use the InnoDB database engine, while the OLAP ones use MyISAM.

Digg.com experiences high popularity and 98% of the time its database is accessed and read.
Maintaining the website’s peak performance remains the focus especially for the Digg engineers amid its growing popularity. The company is finding it difficult to scale up further due to the simple fact that it cannot afford to buy more physical memory. Digg.com was also facing a glitch with its storage misreporting, the status of data synchronization.

The constant challenge that needs to be dealt with is the slow performance caused due to astronomical data tables.