Category Archives: Web development

Avoiding rate limits when polling feeds

One of the cool things about the web is the ability to integrate content from a variety of sources. For example, you might want to pull in news data from an RSS feed and have the news items displayed on your web site.

This is easy to do with a script (such as a program written in Perl, PHP or Python etc) running via a cron job. The script then stores a fragment of content that you can use within your pages.

However, many services limit the number of content requests you can make in a specific time frame. For example, Twitter limit the number of requests a client can make to their service each hour. This prevents the service being swamped. Exceeding this limit, depending on the service, might result in an error or the IP address of your server being temporarily blocked from accessing that service.

Hitting the buffers

Under normal circumstances you probably won’t need to request a feed so many times that you hit the limits yourself. However, imagine the scenario where your web site is hosted on a shared server. You may only be requesting the feed once every two hours but, if other users on the same server are requesting data from the same service at a much higher rate, the combined number of requests might push you over the limit.

In this situation you might find that your requests will fail even though you are making requests at a low frequency.

Creating another source of the feed

If your access to a service is being limited by factors outside of your control then it’s often helpful to find an alternate source of the feed. This is rarely provided by the host service themselves but came be easily set up using Yahoo! Pipes. Yahoo! Pipes are a great way to aggregate, filter and republish data from online sources.

However, at the most basic level, you can set up a simple pipe that requests the RSS feed from the external service and then republishes the data as RSS at a http://pipes.yahoo.com address (see the ‘Get as RSS’ link for your Pipe).

Screenshot of a simple pipe created in Yahoo! Pipes

Once you have created the Pipe you can then request the Pipes version of the RSS feed rather than the original (remembering that you should still keep the rate of requests at a low level).

The result is that the request for the feed comes from the Yahoo! Pipes service and not directly from your script and you can circumvent the issues caused by other users on the same hosting. This is because services like Yahoo! Pipes are intended to be polled and are often white-listed.

The death of email newsletters?

I must admit I was surprised when the Email Standards Project (http://www.email-standards.org/) launched recently. The project aims to explain why Web standards are important for email. To any techie worth their salt the whole idea must grate horribly. After all, the Web is not email and email is not the Web.

I, and I can’t be alone, just want plain text emails rather than the bloated rubbish punted out by the marketing department (because they can and not because they should). I read my mail on a variety of different devices from my mobile to a desktop PC and using a number of different clients from Thunderbird to Pine. I don’t want to waste time/money by downloading HTML email to my phone (by paying for the extra bandwidth) and is my experience of reading an email really enriched by having it rendered in whatever colour/font the sender thinks I need to see?
Continue reading

Losing the 404 File not found error

There are few things that will make your site look more unprofessional than users following a broken link and receiving a 404 – File not found message. For a good developer it’s easy to ensure that a site is free from broken links. If you’re a Linux user you can use ht://check, if you prefer Windows you can use Xenu and Mac users can try BLT.

However, not all Web developers are as diligent (or competent) as you are. I’m sure you choose Cool URIs that never change and, in those very few circumstances where you do have to change a URI, you put in a proper server-side redirect. However, what about those other Web developers, the ones that link to you, how many of them check links after they’ve created them?

What do you do if another developer makes a mess of linking to you so that it causes a 404 Page Not Found error on your site?
Continue reading

Testing times ahead… or maybe not

Web site testing is one of those things, like sticking to the speed limit on motorways, that everyone says they’ll do but, in reality, most don’t. Like speeding you have the potential to get somewhere more quickly but to do so you must take risks that can be very costly.

Almost without fail, testing will be the one area of a project that’s cut by a client if money or time get tight. In a lot of cases a developer will be told “We don’t need external testers, we’ll just get the guys in the office to give it the once over” or worse still “We’ll fix any problems after we launch”. Continue reading