Building a Website on the DigitalNZ API
Introduction
After an eighteen-month spell at Weta Digital I am back at the National Library, or more precisely the hallowed ground of Digital New Zealand.
When I left, Digital New Zealand was a newborn widget. I have returned to find a fully-fledged search service and API suite, with over 100 content partners.
That's one hundred organisations who have signed on and made their digital material searchable through the DigitalNZ APIs. Just wrangling all that disparate data into a searchable common metadata standard is a mind-boggling achievement, let alone the background work required to make such things happen. Having so many organisations on board makes the search API more valuable than ever.
So let's build something with that DigitalNZ Search Records API, and see how it all works.
We could do something client side with javascript, perhaps some kind of play on the DigitalNZ Search Widget.
Or we could do the obligatory google maps mash up, although the search API isn't yet geographically potentised. Hopefully soon.
Or we could just make a website for discovering images and videos, since the vanilla standard http://search.digitalnz.org isn't particularly exhilarating when used for this purpose. Yes, that sounds like fun.
Fast forward a few days and I had what I thought was a pretty reasonable website built on the DigitalNZ search API. There was only one problem.
When I told the DigitalNZ team about it:
They said:
This went on.
At first I couldn't tell what was happening, but eventually I realised that what I thought was so great about my website was really just what was so great about the DigitalNZ APIs, and the team have been living and breathing them for quite some time. They've become accustomed to these miraculous APIs.
But I think that something this awesome has to be shouted about. I knew just what to do.
It will start with a story about how I came to DigitalNZ and built this picture show website with their API. Then there will be this weird conversation between the DigitalNZ team and me that possibly never happened. Last of all, we'll unpick the website, show all of the source code, and hopefully help everyone on their individual ways to doing really cool things with the DigitalNZ API.
And so it begins.
This may reveal that developing with the DigitalNZ APIs is easier than you thought. At the very least, you can surely pinch some code to use in your own projects.
I need to apologise to users of older versions of Internet Explorer, for how the code snippets appear on your screen. For the rest of us they should just overflow the righthand margin in an ugly but perfectly functional manner.
Building a Website on the DigitalNZ API
Use any Language
I have used PHP and MySQL because they're widely known, friendly, and come for free on this $1 per week hosting plan I found. Best of all, PHP won't snipe at you for a hundred minor things that fancier languages will, so you can write code that just gets it done. Oh, and when it hits a bug, it won't print a whole page of barely intelligible stack trace to the screen without giving a clue to where the error even occurred. No, it'll just say something like: "on line 15 in index.php you missed a semicolon" or something like that. It wins.
If you're making a widget — a small "panel" that sits within another website — you will have to use Javascript or Flash, since widgets run inside the user's browser and the options are limited. But if you're making a normal website with the DigitalNZ API, almost any language and database you can think of will be fine.
Of course, if developing on your own computer, PHP comes pre-installed on Mac OS X and Linux, and you can download a perfectly good setup for Windows called xampp. Perhaps best of all, you can google site:php.net <what I need> and you'll almost always find documentation about the function you need, with lots of examples.
Start with a "Hello World"
Before we look at the NZ Picture Show in detail, we'll start with a really simple script that uses the DigitalNZ Search API to display some results and thumbnails for the search_string given in the URL. Here's how it looks in action - and here's the script:
| Notes | Code |
you can get your API key here • retrieve the parameter from the URL • search for sunrises if nothing else • set up the API call • make the API call • dump the results to the screen (disabled) • loop through the results • linking each one to its original • displaying a thumbnail • and a title • |
<?php
$api_key = '<your api key>';
$search_text=stripslashes($_GET['search_text']);
if(!$search_text) { header('Location:'.$_SERVER['PHP_SELF'].'?search_text=sunrise'); exit; }
$api_call = 'http://api.digitalnz.org/records/v2.json?search_text='.urlencode($search_text).'&api_key='.$api_key;
$api_response = json_decode(file_get_contents($api_call),true);
//var_dump($api_response['results']);
foreach($api_response['results'] as $result) {
echo '<a href="'.$result['source_url'].'">
<img src="'.$result['thumbnail_url'].'"/>
<br />'.htmlspecialchars($result['title'],ENT_COMPAT,'UTF-8').'</a><br /><br />';
}
?>
|
One of the great things about PHP is that you can probably copy the code above, paste it into a new text editor window, save it as helloworld.php to the www or htdocs folder on your own computer or on your own webhost, and point your browser at http://localhost/helloworld.php. The only thing you'll need is your own API key, which is just a long string of characters by which the API will know you. Luckily, getting one is free, quick and easy - just follow the instructions at http://digitalnz.org/developer/getting-started/.
If you're wondering how this API call actually works:
$api_response = json_decode(file_get_contents($api_call),true);
file_get_contents is a PHP function that retrieves a URL (just like a web browser, only it puts the output in a string instead of on the screen). Because we want the data in an array (which allows us to use the super handy foreach loop), we then pass it to json_decode. Job done!
Use what you like from the Source Code of the NZ Picture Show
The website comprises 4 PHP files, included in full below:
index.php - the main script
functions.php - a bunch of functions used by index.php
constants.php - just some constants, such as API key, number of results to return, etc
db.php - a small script that adds a suggestion to the database for the 'Popular' section on the front page. Very optional; I just wanted to show MySQL in use.
There's also a simple css file, some javascript (jquery and jquery.infinitescroll, to automatically load more images as you approach the bottom of the page) and a logo. But we're not concerned with them.
Let's look at the php files one at a time
The Source Code, Part I: index.php
The default landing page is called index.php for historic reasons; it doesn't have anything to do with indexing.
Note that any raw html in a php file is just output directly to the browser as html; to invoke php code you need to enclose the code in <?php and ?>.
| Notes | Code |
get the parameters from the url call the API • output the html to set up the page, its search forms and so on. if there's some results... display the 'tag clouds' at the top • display the thumbnails • make a link to the next page • or there's no results... this was a user search • that found nothing or the API returned an error • or this is the landing page so display the 'Popular' cloud • and let them add a Popular term • load the javascript for twitter, and the infinite scroll feature and configure it |
<?php
include 'constants.php';
include 'functions.php';
$search_text=stripslashes($_GET['search_text']);
$applied_filters=stripslashes($_GET['applied_filters']);
$page=max(1,floor($_GET['page']));
$suggestionadded=floor($_GET['suggestionadded']);
if($search_text or $applied_filters) {
$api_response=call_api($search_text,$applied_filters,$page);
if($api_response) $result_count=$api_response['result_count']; else $api_failure=1;
}
?>
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<link rel="stylesheet" type="text/css" href="css/basic.css" media="screen" />
<script type="text/javascript" src="js/showhide.js"></script>
<title>NZ Picture Show | <?php if ($search_text) echo htmlentities($search_text); else echo 'Home';?></title>
</head>
<body OnLoad="document.searchform.search_text.focus();">
<div class="block banner">
<div class="headline"><a href="?">NZ Picture Show</a><br />
<a href="http://twitter.com/share" class="twitter-share-button" data-count="none" data-via="digitalnz">Tweet</a></div>
<div class="search">
<form name="searchform" action="">
<input name="search_text" type="text" class="textbox" value="<?php echo htmlspecialchars($search_text,ENT_COMPAT,'UTF-8'); ?>"><br />
<input class="Search button" value="Search" type="submit">
<?php if ($result_count>0) echo $result_count.' Results found in '.$execution_time.' seconds.
<a target="_blank" href="http://search.digitalnz.org/search?search_text='.
urlencode(DEFAULT_SEARCH_TEXT.' '.$search_text.' '.$applied_filters).'">View on DigitalNZ</a>';
else echo 'Type a word or two and click Search'; ?>
<span onclick="showhide('abouttext'); return true;">About</span> <a href="?">Popular</a>
</form>
<div id="abouttext" style="display:none;">
<div class="h">About this website</div>
This website demonstrates use of the <a href="http://digitalnz.org" target="_blank">
Digital New Zealand</a> API to query and discover Digital New Zealand content. A tutorial, including the
full source code, is available at <a href="http://digitalnz.org" target="_blank"
>http://digitalnz.org</a>.
<div class="h">Terms of Use</div>
Use of this website is governed by the terms of use of DigitalNZ at
<a href="http://digitalnz.org/terms/" target="_blank">http://digitalnz.org/terms/</a>.
</div>
</div>
<div style="float:right;">
<a target="_blank" href="http://www.digitalnz.org"><img style="border:0;" src="images/DNZ_Logo.jpg" alt="Powered by Digital New Zealand"></a>
</div>
<div style="clear:both;"></div>
</div>
<?php
if($result_count>0) {
display_clouds($api_response,$search_text,$applied_filters);
echo '<div id="content" style="clear:both; ">';
display_thumbs($api_response);
echo '</div>';
echo '<div style="clear:both;"></div>';
echo '<div class="navigation"><a href="?search_text='.urlencode($search_text).'&applied_filters='.urlencode($applied_filters).'&page='.($page+1).'">Next</a></div>';
} else {
if($api_response) echo '<div class="block"><span class="heading">NO RESULTS</span>
<br /> <br />Sorry, your search returned no results. Please try again, or click a suggestion below to get started.</div>';
elseif($api_failure) echo '<div class="block"><span class="heading" style="color:red;">ERROR</span>
<br /> <br />Sorry, your search returned an error from the search service. Please try another search.</div>';
echo '<div class="block"><span class="heading">POPULAR</span>';
display_popular();
echo '</div>';
echo '<div class="block"><span class="heading">SUGGEST A SEARCH TERM</span><br /> <br />
'.($suggestionadded?'Thanks! Your suggestion has been queued for moderation. ':'').'Add your favourite search to the Popular section:
<form name="suggestionform" action="db.php">
<input name="suggestion" type="text" value=""><br />
<input class="Search button" value="Add" type="submit">
</form>
</div>';
}
?>
<script type="text/javascript" src="http://platform.twitter.com/widgets.js"></script>
<script type="text/javascript" src="js/jquery-1.3.2.min.js"></script>
<script type="text/javascript" src="js/jquery.infinitescroll.min.js"></script>
<script type="text/javascript">
$('#content').infinitescroll({
navSelector : "div.navigation",
nextSelector : "div.navigation a:first",
itemSelector : "#content div.post",
bufferPx : 100,
loadingImg : "images/blank.gif",
loadingText : "",
donetext : ""
});
</script>
</body>
</html>
|
The Source Code, Part II: functions.php
This contains the functions used to render the search results and facet tag clouds.
| Notes | Code |
call_api Calls the Search API and returns the results All these API parameters are documented here. The actual API call If there's an error, return false tag_size returns a font-size % sanitise takes care of macrons etc display_clouds displays a tag cloud for each facet in the results the facets in the array can best be seen with a var_dump from the api, but they are returned as an array of facets, where each facet is made up of a facet_field (eg decade) and an array of values, where each value has num_results (the number of results that would be returned if that facet were applied) and the string (eg "1960-1969") array_multisort puts the results in alphabetical order within each facet loop through the tag_fields, tag_values and tag_counts arrays Add a string to the search to filter the results based on the tag clicked, eg decade:1960-1969 display_thumbs Displays a page of thumbnails with captions beneath display the thumbnail • the pin • the title • and the content provider • display_popular Display a tag cloud of popular options Retrieve all the 'published' suggestions • Print them out at the appropriate size • |
<?php
function call_api($search_text,$applied_filters,$page) {
global $execution_time;
$api_call='http://api.digitalnz.org/records/v2.json?num_results='.NUM_RESULTS.
'&search_text='.urlencode(implode(array(DEFAULT_SEARCH_TEXT,$search_text,$applied_filters),' ')).
'&facets='.SHOW_FACETS.
'&facet_num_results='.FACET_NUM_RESULTS.
'&start='.max(0,($page-1)*NUM_RESULTS).
'&sort='.SORT_KEY.
'&direction='.SORT_DIRECTION.
'&api_key='.API_KEY;
$time_start = microtime(true);
$result=json_decode(file_get_contents($api_call),true);
$time_end = microtime(true);
$execution_time = round($time_end - $time_start,2);
if(is_array($result)) return $result; else return false;
}
function tag_size($tag_count,$max_count) {
return min(TAG_MAX_SCALE,floor(TAG_SIZE*(1.0+(TAG_RATIO*$tag_count-$max_count/2)/$max_count)));
}
function sanitise($text) {
return htmlspecialchars($text,ENT_COMPAT,'UTF-8');
}
function display_clouds($api_response,$search_text,$applied_filters) {
$max_count=0;
foreach($api_response['facets'] as $facets) {
$facet_field= $facets['facet_field'];
foreach($facets['values'] as $value) {
$facet_value=$value['name'];
$facet_count=$value['num_results'];
$tag_values[]=$facet_value;
$tag_fields[]=$facet_field;
$tag_counts[]=$facet_count;
$max_count=max($max_count,$value['num_results']);
}
}
array_multisort($tag_fields,$tag_values,$tag_counts);
$c=count($tag_values);
echo '<div class="block">';
for($i=0;$i<$c;$i++) {
# Add Tag Category heading if it's the first tag of its category
if ($tag_fields[$i]<>$last_tag_field) { if($last_tag_field) echo '</div><div class="block">';
echo '<span class="heading">'.strtoupper($tag_fields[$i]).'</span>'; $last_tag_field=$tag_fields[$i]; }
if(!(strpos(urlencode($applied_filters),$tag_fields[$i].'%3A%22'.urlencode($tag_values[$i]).'%22')===FALSE)) {
# Tag has already applied
echo '<a class="tag'.($i%2).'" title="remove '.htmlspecialchars($tag_fields[$i].':"'.$tag_values[$i].'"',ENT_COMPAT,'UTF-8').'
from filters" style="font-weight:bold; color: #c8470f; font-size: '.tag_size($tag_counts[$i],$max_count).'%;"
href="?search_text='.urlencode($search_text).'&applied_filters='.str_replace($tag_fields[$i].'%3A%22'.
urlencode($tag_values[$i]).'%22','',urlencode($applied_filters));
} else {
# Tag has not been applied
echo '<a class="tag'.($i%2).'" title="'.$tag_counts[$i].' items" style="font-size: '.tag_size($tag_counts[$i],$max_count).'%;"
href="?search_text='.urlencode($search_text).'&applied_filters='.urlencode($applied_filters).
($applied_filters?'+':'').$tag_fields[$i].'%3A%22'.urlencode($tag_values[$i]).'%22';
}
echo '">'.str_replace(' ',' ',$tag_values[$i]).'</a> ';
}
echo '</div>';
}
function display_thumbs($api_response) {
foreach($api_response['results'] as $value) {
echo '
<div class="post">
<a target="_blank" href="'.sanitise($value['source_url']).'">
<img src="'.sanitise($value['thumbnail_url']).'"
title="'.sanitise($value['title']).' ('.sanitise($value['content_provider']).'): '.sanitise($value['description']).'"/>
</a>
<div class="metadata">
<a target="_blank" href="http://search.digitalnz.org/records/show/'.$value['id'].'.html">
<img class="pin" src="images/pin.gif"/>
</a>
<div>
<a title="'.sanitise($value['title']).'" target="_blank" href="'.sanitise($value['source_url']).'">'.
substr(sanitise($value['title']),0,45).'</a>
</div>
<div>
<a class="provider" title="'.sanitise($value['content_provider']).'" target="_blank" href="'.
sanitise($value['source_url']).'">'.substr(sanitise($value['content_provider']),0,45).'</a>
</div>
</div>
</div>';
}
}
function display_popular() {
$connection = mysql_connect('localhost', MYSQL_USER, MYSQL_PASSWORD);
$max = mysql_fetch_row(mysql_query('select max(hitcount) from younge_dnz.suggestions where published=1'));
$max_count=$max[0];
$suggestions = mysql_query('select suggestion,hitcount from younge_dnz.suggestions where published=1 order by suggestion');
$i=0;
while($row=mysql_fetch_row($suggestions)) {
$suggestion=$row[0];
$hitcount=$row[1];
echo '<a class="tag'.($i%2).'" title="'.$hitcount.' items" style="font-size: '.tag_size($hitcount,$max_count).'%;" href="?search_text='.
urlencode($suggestion).'">'.str_replace(' ',' ',$suggestion).'</a> ';
$i++;
}
}
?>
|
The Source Code, Part III: constants.php
This one's simple enough. A bunch of things you can configure.
| Notes | Code |
your API key • how many thumbnails to grab at once • the filter that is applied to all the searches • how many tags are in each cloud • how the results are sorted - a field• how the results are sorted - asc/desc • database username • database password • which tag clouds to show • tag cloud largest font % scale • tag cloud overall font size • tag cloud size differential • |
<?php
define('API_KEY','put your key here');
define('NUM_RESULTS',50);
define('DEFAULT_SEARCH_TEXT','(category:Images OR category:Videos)');
define('FACET_NUM_RESULTS',20);
define('SORT_KEY','date');
define('SORT_DIRECTION','asc');
define('MYSQL_USER','put your database username here');
define('MYSQL_PASSWORD','put your database password, if any, here');
define('SHOW_FACETS','decade,placename,rights,category,content_partner');#choose from category,century,decade,year,creator,rights,collection,content_partner
define('TAG_MAX_SCALE',250);
define('TAG_SIZE',200);
define('TAG_RATIO',1.6);
?>
|
The Source Code, Part IV: db.php
Takes a script parameter and adds it to the MySQL database of popular searches, if valid. Very optional.| Notes | Code |
get the parameter from the URL • if there's a suggestion and it's not too long • connect to the database • see how many results this search would return • if it would return more than one result • add it to the database • and flag the fact that we added it • redirect back to the home page • |
<?php
include 'constants.php';
include 'functions.php';
$suggestion = stripslashes($_GET['suggestion']);
if($suggestion<>'' and substr_count($suggestion," ")<4) {
$connection = mysql_connect('localhost', MYSQL_USER, MYSQL_PASSWORD);
$api_response=call_api($suggestion,'','',0,0);
$result_count=$api_response['result_count'];
if($result_count>0) {
mysql_query('insert into younge_dnz.suggestions (suggestion,hitcount,published) values ("'.mysql_real_escape_string(ucfirst($suggestion)).'",'.$result_count.',0)');
$suggestionadded=1;
}
}
header("Location: http://".$_SERVER['HTTP_HOST'].rtrim(dirname($_SERVER['PHP_SELF']), '/\\')."/?suggestionadded=".$suggestionadded);
exit;
?>
|
I'm using a MySQL database to store the popular searches that appear on the homepage. Like PHP, MySQL is free and ubiquitous. The table can be created by executing the following SQL, using PHPMyAdmin or similar:
| Notes | SQL |
the key words • how many hits this search gets • whether this search is visible • |
CREATE TABLE `suggestions` ( `suggestionid` int(10) unsigned NOT NULL AUTO_INCREMENT, `suggestion` varchar(255) NOT NULL, `hitcount` int(10) unsigned NOT NULL DEFAULT '0', `published` tinyint(1) NOT NULL DEFAULT '0', PRIMARY KEY (`suggestionid`), UNIQUE KEY `suggestion` (`suggestion`), KEY `published` (`published`) ) ENGINE=MyISAM AUTO_INCREMENT=132 DEFAULT CHARSET=latin1 |
The Wrap Up
The DigitalNZ APIs open up an unbelievably rich and deep content source for you to use in your own projects.
As you've seen here, you can use the APIs to create something new, and then share that website or widget with the world.
Using the same techniques, you can also combine our APIs with a growing number of other organisations' APIs, to create a "mash up" (or something so cool it doesn't even have a name yet) that is greater than the sum of its parts. This is a very exciting area for development.
Whatever you're doing, or thinking about doing, with the DigitalNZ APIs, I hope you've found something useful in this post, and I look forward to seeing what you've made! Please send me a link, or post a comment below.
Elliott Young
DigitalNZ
Add new comment
Showing 10 comments
Seriously cool stuff. How long until "geographic potentisation", just out of interest?
Just saw this post via Twitter... looks cool thanks for the info Elliott
Hi Virginia,
The two things we still need are:
* return lat/long in the Search results API
* allow searching by location (eg localsolr)
Can't say for sure, but hopefully not too long. Want!
In the meantime, people can geotag stuff, and retrieve the latitude and longitude of individual results with the Get metadata API.
Great story! And I like the resulting app as well!
Thanks Paul, David, glad to hear it's useful.
I've now expanded the results from just images and videos to include the 35,000 "reference sources" (most of which have images and other valuable info) by changing the default search text in constants.php:
define('DEFAULT_SEARCH_TEXT','(category:Images OR category:Videos OR category:"Reference sources")');
I've added some basic mapping functionality to the NZ Picture Show (http://elliottyoung.com/labs/nzpictureshow/) and blogged about it (http://digitalnz.org/blog/news/article-api-enhancement-brings-geotagging-to-search-results).
So to see the exact website covered in this blog post go to http://elliottyoung.com/labs/nzpictureshow/v1/
Nice explanation , easy to follow and quite a grunty api !
oh, and a great app :)
Great website. The DigitalNZ APIs open up an unbelievably rich and deep content source for you to use in your own projects. As you've seen here, you can use the APIs to create something new, and then share that website or widget with the world.
http://howtomusclebuild.eliminarlagrasa.com/
In the meantime, people can geotag stuff, and retrieve the latitude and longitude of individual results with the Get metadata API.
http://comoperderdepeso.eliminarlagrasa.com/