Yahoo BOSS API

Yahoo BOSSYahoo recently released a new search API. Known as Yahoo BOSS, for “Build your Own Searcg Service,” the API allows you to query search results from their servers, format them however you want, mash the data up with other services, and even re-order results. You get “Unlimited*” queries (they just reserve “the right to limit unintended usage, such as automated querying by bots”) and they don’t even require attribution.

Even though I’m what you could possibly call a “Google/Apple fanboy” (though you would be advised to not say such things…), and I’ve long dismissed Yahoo as boring, geared towards web newbies, among other things, I have to admit, this is a great API. Google never gave us anything like this (despite their seemingly unlimited resources) and they discontinued their fairly limited search API. (As a side note, I also admit that Yahoo owns some great web services, such as Flickr and Del.icio.us.)

I’ve already got to work playing with the API, creating a sort of search mashup. I figured I’d share a little bit of code, and show you how to create a basic SERP. Be warned, the following requires PHP5 and some cURL black magic. (If you have no idea what I just said, read a book, and come back later.)

In order to use Yahoo BOSS, you need to get an API key. If you have a Yahoo account, it will only take about five minutes. Once you’ve got a key, you may want to take a quick look at the API documentation. There aren’t any functioning examples or tutorials, but the guide tells you what sort of queries you can send, and what you can expect to get back in response.

Now for the fun part. Here’s a basic example I threw together. You can use it as a starting point for your own project. It’s fairly basic, as it’s main goal is to illustrate how to go about requesting data from the api and displaying it. You should be able to run the code as-is, though you’ll need to insert your API key before running the script:

<?php if ($_GET['s'] == '') { ?>
<form method="get" style="margin-bottom:30px; margin-top:20px;">
<input type="text" name="s" size="30" /> <input type="submit" value="Search" />
</form>
<?php } ?>

<?php
if ($_GET['s'] != '') {

//Gather data and prepare query
$thequery = urlencode($_GET['s']);
$yhost = 'http://boss.yahooapis.com';
$apikey = 'PASTE_YOUR_API_KEY_HERE';
$url = $yhost.'/ysearch/web/v1/'.$thequery.'?appid='.$apikey.'&format=xml';

//Get the results
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
$results = new SimpleXmlElement($data, LIBXML_NOCDATA);

//echo the results
foreach ($results->resultset_web->result as $theresult) {
echo '<a href="'.$theresult->clickurl.'">'.$theresult->title.'</a><br/>';
echo $theresult->abstract.'<br/>';
echo '<small><i>'.$theresult->dispurl.'</i></small><br/>';
echo '<br/><br/>';
}

}
?>

As you can see, this is about as basic as an example as you can get. It shows the first ten web results for whatever you enter in the box. If you were to append “&start=20″ to the end of the query URL in the $url variable, it would show results 11-20 instead. That’s the beginning of creating a pagination system to allow a user to move beyond page one of the results.

If you have a little bit of PHP experience (like I recommended earlier; weren’t you listening?), the code should be fairly self-explanetory.

  1. First, the script outputs a search box, unless the form has already been submitted. If that’s the case, the script will move on two step 2.
  2. If someone has submitted the form, the value of the “s” variable from the browser URL’s query string (it looks like “?s=blah”) is reformatted so spaces and other special characters won’t make the BOSS API puke when it sees them.
  3. The API query is assembled. If you were to place the word “echo” before the “$url=…” statement you’d get to see the complete URL when the result page loads. It looks something like “http://boss.yahooapis.com/ysearch/web/v1/iphone?appid=YOUR_API_KEY&format=xml”, where iPhone is the search query and YOUR_API_KEY is, well, your API key. If you’re a JSON maniac you can replace format=xml with format=json, but if you want to do that, you probably don’t need my tutorial.
  4. Next, we use the useful cURL extension to send a request to Yahoo saying “give us the data, Yahoo.” After it gets a response, the SimpleXmlElement line parses the returned XML file into a PHP Object.
  5. Now we move onto the part where we display the results with a simple foreach() loop. You can style it however you want, maybe write some more semantically correct XHTML, but good old <br /> works well enough to illustrate my point. (Yeah, I’m lazy. :D )

Now make it your own. You’ve got a basic starting point, and hopefully some knowledge on how it works. Do something unique with it, mash the data up with data from another API.

EDIT: Some of you have been having trouble with the code above. WordPress seems to be trying to “fix” parts of it. Instead of copy/pasting the code above, you should download this .txt file.

  • sundeep

    thanks a lot. This was precisely what i was looking for ..

  • jash

    hey I’m a newb (but quick learner) with a quick question…the code turns up an error for the variable s as undefined!should this happen with the current code above?

  • http://www.ruelicke.net Marco Ruelicke.net

    jash:
    have a look at the form you use to submit the search query. I’d guess it isn’t named “s” (html attribute: name=”s”) but something else. You may have to change the “s” in the code above or the name of your input field. :)

  • jash

    copy that!Ok past that I have a parsing error with the url string…wt…It’s hiccup is at the colon ‘:’double quotes?any ideas?

  • jash

    I believe this line is an issue for me…$yhost = ‘http://boss.yahooapis.com’;where the // is commenting out the string

  • nhoss2

    im having an error.. it says Parse error: syntax error, unexpected ‘:’ in search.php on line 12.. i put in my ID and everything..

  • http://www.ruelicke.net Marco Ruelicke.net

    convert // to // (that’s backslash slash backslash slash)

    about the colon: have a look at line 12. If the colon is at the end of the line, try replacing it with a semicolon.

  • jash

    I suspect that if the id was wrong you’d get a 403 back from y!that string is driving me crazy….Matt, are you there bro?

  • http://www.webmaster-source.com Matt

    @Marco. Thanks for helping to answer questions. I’m getting some emails about this post, and even more about my wp125 plugin, so it’s nice to not be the only “tech support guy.” :D

  • http://www.ruelicke.net Marco Ruelicke.net

    no problem, Matt, it is entertaining me to tinker around with some coding problems. Let me know if there is anything else I can help you with :)

    jash, maybe you can show us the source as it helps us to find the exact cause of the problem :)

  • jash

    Sure Marco, And thanks for your time here’s current source beginning:<?php if ($_GET['index'] == ”) { ?>    <form method=”get” style=”margin-bottom:30px; margin-top:20px;”>    <input type=”text” name=”index” size=”30″ /> <input type=”submit” value=”Search” />    </form>    <?php } ?>    <?php    if ($_GET['index'] != ”) {    //Gather data and prepare query    $thequery = urlencode($_GET['index']);    $yhost = ‘http://boss.yahooapis.com’;    $apikey = ‘mykey’;    $url = $yhost.’/ysearch/web/v1/’.$thequery.’?appid=’.$apikey.’&format=xml’;where the $yost line is returning an error for the string…just for kicks I took out the http:// and that seem to go away althoughI was unable to see the url build because of another string error with the$url variable….now it doesn’t like the = assignments in the appid and format!So it seems we have a problem with building the url!  right?

  • http://elksy.com nhoss2

    i fixed everything, the whole thing was stuffed up because the encoding was ANSI instead of unicode.. but now when i got to it, it doest really work because 1. there is no submit button and 2. it shows some of the code at the bottom.. my sites here: http://www.elksy.com

  • jash

    ok…it’s safe to say that the btn is no problem but the other code looks likeanother comment issue!  Something not closed?  Could you post your code nhoss2 when you get a chance?  We are making progress!  :)

  • http://elksy.com nhoss2

    <?php if ($_GET['s'] == ”) { ?><form method=”get” style=”margin-bottom:30px; margin-top:20px;”><input type=”text” name=”s” size=”30″ /> <input type=”submit” value=”Search” /></form><?php } ?><?phpif ($_GET['s'] != ”) {//Gather data and prepare query$thequery = urlencode($_GET['s']);$yhost = ‘http://boss.yahooapis.com’;$apikey = ‘THE KEY’;$url = $yhost.’/ysearch/web/v1/’.$thequery.’?appid=’.$apikey.’&format=xml’;//Get the results$ch = curl_init($url);curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);curl_setopt($ch, CURLOPT_HEADER, 0);$data = curl_exec($ch);curl_close($ch);$results = new SimpleXmlElement($data, LIBXML_NOCDATA);//echo the resultsforeach ($results->resultset_web->result as $theresult) {echo ‘<a href=”‘.$theresult->clickurl.’”>’.$theresult->title.’</a><br/>’;echo $theresult->abstract.’<br/>’;echo ‘<small><i>’.$theresult->dispurl.’</i></small><br/>’;echo ‘<br/><br/>’;}}?>

  • http://www.ruelicke.net Marco Ruelicke.net

    Jash, maybe you can put the code into a txt which I can download. In the comments here it gets messed up :)

    nhoss2, same for you, if you can show us the code, we have it easier with finding a solution :)

  • jash

    Thanks Marco, I saw that…link: here

  • http://elksy.com nhoss2

    done yeah, im having a lot of trouble with line 22 which is:

    $results = new SimpleXmlElement($data, LIBXML_NOCDATA);

  • http://www.ruelicke.net Marco Ruelicke.net

    hm…so far I discovered a common problem:

    you just copy and pasted the code, but the code posted here is not ok, as the quotation marks are converted. Go through your code and replace the quotation marks or download the txt Matt added to his post. View as *.phps.

    About the other problem, make sure you have PHP5 as the code only works with PHP5. Working Example.

  • jash

    PHP5 verified…corrected the quotes but now I’m having another error:Uncaught exception ‘Exception’ with message
    ‘SimpleXMLElement::__construct() expects exactly 1 parameter, 2 given’
    in /var/www/html/domain.com/index7.php:34
    Stack trace:
    #0 /var/www/html/domain.com/index7.php(34):
    SimpleXMLElement->__construct(‘/var/www/html/domain.com/index7.php
    on line 34curl and simplexml are enabled!

  • http://www.ruelicke.net Marco Ruelicke.net

    source code?

  • http://elksy.com nhoss2

    it works! thank you soo much!! (site: http://www.axisgames.co.cc/search.php) wow this is great!! is there any way to hide the click url?

    its also a bit glitchy (thats because of the bad server)
    and how do i go to the next page?

  • http://www.ruelicke.net Marco Ruelicke.net

    I didn’t read the api guide, yet, you may want to check it out to find a way to have search pages (link is in Matt’s article) :)

    Which click url do you mean? If you mean the link, then just remove the echo line with the link.

    echo '<a href="'.$theresult->clickurl.'">'.$theresult->title.'</a><br/>';

  • http://www.webmaster-source.com Matt

    @nhboss2 To hide the URL, you want to remove the echo '<small><i>'.$theresult->dispurl.'</i></small><br/>'; line. Just a quick definition: The clickurl is the URL that the result link must point to (Yahoo keeps click data, and they demand that it be used for clickthroughs). Dispurl is the URL formatted for display. And there’s also a plain “url” for usage in other parts of the script.

  • jash

    Marco or Matt, Here’s the source…thanks!

  • vince

    hi thank you for this post it is what i was searching for…(more or less)what is missing here is:
    1-next and previous pages
    2-Suggest terms
    3-Site (domain)search
    4-top queries (if possible)

    I found this site where it is explained how to do The first 3 but it is in phpcake…here is the link i ll try to find out how to do these on my own. I ll Share it with you if i manadge to do itheve a nice day

  • pluto

    yeah it would be great if you could add these features :D

  • http://www.aboriginalaus.com/bossy/yboss.php kanedogg

    Dudes im having the same issues with the XML error all my code or “your code seems fine …error:Fatal error: Uncaught exception ‘Exception’ with message ‘String could not be parsed as XML’ in D:\Hosting\2943519\html\bossy\yboss.php:32 Stack trace: #0 D:\Hosting\2943519\html\bossy\yboss.php(32): SimpleXMLElement->__construct(”, 16384) #1 {main} thrown in D:\Hosting\2943519\html\bossy\yboss.php on line 32
    Code  linked here :http://www.aboriginalaus.com/bossy/yboss.php
    Can someone assist please ?! my provider has CURL/PHP5 all switched on ??

  • http://www.ruelicke.net Marco Ruelicke.net

    I’m still trying to reproduce this kind of error, so far I failed :S

  • http://www.aboriginalaus.com/bossy/yboss.php kanedogg

    Marco is seems something to do with the line 32 and that line is the simpleXmlElement ………. Is there another way i could produce the results returned other than Xml thats as simple to code as i’m a NOOB with PHP>?
    Also could it be the fact its an external address request ? Or that it cannot decode the results to XML ?
    dunno just a thought ?>!

  • Sonny

    Great work!  I was able to run it on my local Windows XP machine with WAMP installed.  I ran into problems since my php_curl extension is not activated by default. I had to manually activate it.  It works now.
    I have a few questions hope some of you guys can help.
    1.  How to increase the actual search results.  The default only gives me 10 results per query.
    2. Add Numbers or “Next” & “Previous” links to navigate thru the results.
    3. Get actual totals of the the search results.
    Thanks.
     
     

  • http://www.webmaster-source.com Matt

    @Sonny, Yahoo limits the amount of results per query to ten. There is no way around this save querying twice and merging the results. To answer your next two questions, have a look at the documentation: http://developer.yahoo.com/search/boss/boss_guide/. The anwers to both lie within. (I could have shown how to do pagination in my example, but I wanted to keep it as simple as possible.)

  • Sonny

    Thanks Matt.  I was able to get the next results (page) by changing the start field to 11 since 10 was the last record in my first page.  I was able to check BOSS results against yahoo results and verified that the results were close to equal.  I manually updated the “start” field value to do this.  I believe I can work around the number of resullts per query per page through the next pages links since I can pass the query value and the “start” field value throught the number page links.  Next step I’ll do now is to pass the “start” field value through the number links and determining the number of page links (dependent on the “deeephits” value -  thanks for the documentation tip).  I am having trouble how to declare/use/display “deephits” field value in my code.  I am still a newbie when it comes to php programming.  Let me know if I am on the right track of coding this and if you have other suggestions how to go about doing this.  Thanks again.

  • http://www.webmaster-source.com Matt

    @Sonny, You’re definitely on the right track. If you’re having trouble getting deephits, I’d recommend trying this: echo ‘<pre>’; print_r($results); echo ‘</pre>’; It visually shows the result set, so you can construct your object calls better.

  • Sonny

    Thanks Matt for the prompt response.  This will greatly help.  I am very excited with this project and I would like to explore the capibilities.  I have a few more questions though, hope you don’t get tired answering.  I am just confused on how will I go about pagination.  I am not sure whether the results being presented during every call (through the change of the “start” field value in the pagination numbers) is referenced from the “deephits” count or from the “totalhits” count.  The totals of each greatly differ and I don’t want to end early the number of succeeding pages knowing there is more data to display.  Do you know which to reference?  Another one is with the result set (thanks for the code on echoing the array results).  It seems that the “clickurl” field is explicitly referencing a link to yahoo and is also visible at the status bar during “mouseover” events prior to clicking the link item.  I believe that this should not be as Yahoo mentioned in their TOU that we must not reference them in anyway when we rollout our developed products.  I would like to make my application identified as my product which by the way was the reason and thrust of why Yahoo BOSS was made in the first place.  For developers.  I understand that this is the process Yahoo! Boss use to track our AppID’s. I just dont know why can’t we do it the way they do it with their site. I have an idea but not sure if possible.  Can javascript and php code be interused into one file?  I intend to mask status bar to display the “dispurl” field by using javascript. Not sure if this will also work across all browsers and if their Javascript is disabled. Let me know your thoughts. Thanks.

  • http://zebradezign.com Liz

    Hello,
    Having trouble with “$results = new SimpleXmlElement($data, LIBXML_NOCDATA);”….is there a work around for this?Also, it was mentioned in an earlier post to use PHP5…..is it suggested to use an editor with PHP5 or make sure the web host server is up to date with PHP5? (perhaps both). 

  • http://zebradezign.com Liz

    Hello All,
    Just posted a request for help with the a”$results = new SimpleXmlElement($data, LIBXML_NOCDATA);”… my solution, and it could be yours too, make sure the web server you’re hosting the file from has PHP5 enabled. It took care of my difficulties.

  • kanedogg

    hasn’t fixed mine and my HOST is PHP5 only !! so figure that one?? the post is above. the error message says something about the simplexmlElement being build wrong, like it needs more prefs in it or alike ???

  • http://zebradezign.com Liz

    Hi kanedogg,
    I was having same trouble with the simplexmlElement too. What I did, which seemed to work, is copy over that entire segment with a successful segment that can be found here on this page…just follow one of the links to see and copy the code. I ‘saved as’ a php file then uploaded to my server. That seemed to clear up whatever bug I had with my program after making sure my server was running php5.

  • http://zebradezign.com Liz

    Just finished my take-off of the provided boss search engine script….check it out and let me know your thoughts. I am at the point of adding the prev and next page tags….not sure how to do this, yet. review my search form

  • kanedogg

    Peoples could someone please give me another example of how i could return the results ?? instead of the simpleXMLElement.
    I would like to see if i can get another way working, maybe even a print_r or Return? i’m unsure of php as i dont know it at all. Thanks in advance.
    Kanedogg

  • http://zebradezign.com Liz

    Kanedogg…I hear your pain, I’ve been where you are now. I think you might find the Yahoo developers kit most helpful. The kit comes with easy to use and understand examples and templates. You can view the one I produced here, it is different than the example code presented here. Good Luck and understand there are answers, just not always easy to find.

  • kanedogg

    Liz ,
    i see your page its nice. I cannot see the source though! it would be nice…..can you show me it please ?in a txt or alike.
    thanks in advanceKanedogg

  • http://zebradezign.com Liz

    Kanedogg…. I’ll have to figure out how to submit the code without having it look like a mess.

  • Pingback: Sikbox - Live Search Made Easy | Webmaster-Source

  • Vishal

    Hell guys,

    This is very good script for enable search features on our website. But i m facing a problem about pagination. How it will implemented. Actually i want only total number of results from yahoo. so that i can do the pagination script. My problem is i dont know, how many records in result. Thats why i do not forward to code on pagination. It only mentioned that ‘start’ parameter, for passing values to ‘start’ parameter. I should have to number of records in result set. Please help me on this. So that i can move forward on development.

  • http://www.aboriginalaus.com Kanedogg

    Vish,
    i hope i understand your question ? BUT you only want to grab the TOTAL amount of returned results from your search? Right?
    Man have you ever thought to dig into the returned results in any way?? i have thrashed my way through it & theres a node in the resultset called [totalhits="246994"] & another called [deephits="8170000"] these should be what your looking for bud!

    You could grab this value and implement your own pager with pagination methods i would think, Hope it helps.

    Cheers

  • Pingback: Yahoo Boss PHP code examples are hard to come by : SubscribeToMyBlog.com

  • http://www.mistix.co.uk mark

    This is great thanks. I was looking for a simple site serach without showing competing ads. Your script and the Yahoo API works great. I changed line 15 to:

    $url = $yhost.'/ysearch/web/v1/site%3Ayoursite.com+'.$thequery.'?appid='.$apikey.'&format=xml';

    to enable results for my site only.

  • trupti

    this example is not working for me.$data shows (bool) false value to me.Can u explain the reason.