Screen Scraping Web Sites with QlikView

There are websites that essentially provide a web interface to a database. Data is often presented as a HTML table, sometimes spread on multiple pages. When you feel limited by the slicing and dicing options as well as search capability, it is tempting to pull the data into QlikView. However most websites are designed for end users and do not provide bulk export. QlikView does provide a HTML parser than can read HTML tables directly from a web site.

First you have to make sure that the interesting data is in HTML tabular format by looking at the source of the page. HTML tables use the <table><tr><td></td></tr></table> structure. Next you should observe the URL in the browser address bar as you click through the website, noticing the differences when you select different options.

Here is an example of a load script that will load 150 technology jobs from dice.com:

Jobs:
LOAD [Job Title],
     Company,
     Location,
     [Date Posted]
FROM
[http://www.dice.com/job/results/10013?caller=advanced&n=50&src=19&x=all&p=z]
(html, codepage is 1252, embedded labels, table is @1);

for i = 50 to 150 step 50
LOAD [Job Title],
     Company,
     Location,
     [Date Posted]
FROM
[http://www.dice.com/job/results/10013?caller=advanced&n=50&o=$(i)&src=19&x=all&p=z]
(html, codepage is 1252, embedded labels, table is @1);
Next

Notice the $(i) embedded into the second load FROM URL which tells dice.com web server the record number to list from. This is required because the server will list a maximum of 50 records at a time.

Note that most web sites do not allow you to redistribute their content, and some will limit number of hits per day from an IP address. It is a good idea to dump into QVD or CSV after running the load, and using local files for successive loads.

store Jobs into jobs.csv (txt);