Guide - Creating a URI Template Answer Generator

Overview

In this guide, we're going to create an Answer Generator that outputs a link to Yummly. The Answer Generator we create is going to activate in the appropriate context, and will have a user preferences page allowing the user to specify his dietary restrictions and preferences.

Why isn't a Google search good enough? Most recipes on Yummly are found by using its search form. Google may not be able to find all of the recipes, because it only indexes pages is can discovery by crawling. (Put in a technical way, recipe results are in the deep web.) When looking for a recipe, you get the most complete and relevant results by going to multiple recipe sites and using their search forms. Also Google doesn't yet know about your dietary restrictions and preferences so it can't filter out results for you. But Yummly can apply those preferences, and we're going to make it easier to get to the most relevant results.

You may be wondering, "what's the point of adding better links to Yummly when the user can just go to Yummly directly and search there?". Here's a few reasons:

  • The user may not know about or remember to use Yummly, and web search results might not have Yummly in the top results. Having a Yummly link included on the user's page, in the right circumstances, is a good way to keep the user coming back to Yummly.
  • If the user is already using Solve for All, he can just enter his query here, without opening another browser tab. If he enters his query on Solve for All, he'll also get links to other recipe sites.
  • If the user goes to Yummly directly (or gets there with a non-optimal link), she will have to re-enter her dietary restictions and preferences in Yummly's search form before her first search. With Solve for All, her preferences will be already encoded in the link to Yummly, so it will show results that already take into account her dietary restrictions and preferences.
  • An Answer Generator especially built for Yummly can be added to a meta-search engine (called an Engine) for recipes. Recall that Engines are collections of configured Answer Generators and Feeds suitable for a specific domain. You or another user can create an Engine that outputs links to top recipe sites, and possibly searches new recipes that are broadcast via RSS (see Epicurious's RSS Feed for an example). A user of such an Engine would be able to find recipes on multiple sites, just by entering his query a single time. And the link to Yummly would be optimal.

Of course, Yummly is just an example. There are many sites that have data in the deep web, and you can apply this guide to making Answer Generators for them.


Preliminaries

Before we dive in, let's make sure we aren't wasting our time and that what we want to do is feasible. First let's check if an Answer Generator for Yummly is available. Go to Solve for All's plugin search page and enter "yummly". There's one already there, and if you go to its detail page you can try it out (look for the Try it out section). Enter the query hummus and hit Enter. You should be taken to the Solve for All answers page that has a quick link to Yummly. If you click on the hamburger icon, and click Settings ... in the dropdown menu, you should see a dialog box that includes the Preferences tab. Click on that tab. Now you'll see a form with options for Yummly, like dietary restrictions and preferences. At this point, you would probably just add this Answer Generator instead of creating your own (unless this Answer Generator didn't do everything that you wanted). But this Answer Generator was the result of creating this guide, so let's continue!

Finally, let's see how search on Yummly works. If you go to yummly.com and create an account, you'll be able get search results on page like this. On the left side, you'll see options you can apply to filter your results. For me, I chose the following options:

  • Dislike salty food (actually I love salty food, but I just want to cut back)
  • Really dislike bitter food
  • Like spicy food
  • Allergic to seafood (really hard for an Asian, by the way)
  • Vegetarian diet (because I aspire to be, not because I am)
  • Carbs capped at 70 g/serving

When I apply my options, the URL changes to:

http://www.yummly.com/recipes?q=hummus&noUserSettings=true&allowedIngredient=&excludedIngredient=&maxTotalTimeInSeconds=Any+time&flavor.salty=Dislike&flavor.savory=+&flavor.sour=+&flavor.bitter=Really+Dislike&flavor.sweet=+&flavor.spicy=Like&allowedDiet=387%5ELacto-ovo+vegetarian&allowedAllergy=398%5ESeafood-Free&nutrition.cholesterol=+&nutrition.fat=+&nutrition.calories=+&nutrition.carbs=70+g%2Fserving&imagesOnly=false&blogsOnly=false&myRecipesOnly=false&sortBy=relevance

So it looks like there are useful parameters in the URL that aren't the search query. If the URL only had the search query, you could maybe auto-discover an Answer Generator by going to the Answer Generators page in the customization section of Solve for All, and selecting "Create a new Answer Generator".

Now let's check that direct links with these URLs will work. Copy the the Yummly URL, open a new browser tab, and paste the copied URL into the address bar. We see that the same results are displayed and our preferences have been saved. So it appears Yummly is respecting REST conventions and it can be used to create a URI Template Answer Generator. Some sites that require a POST request to get search results, or don't apply the same filters when you do a standalone GET request. But Yummly is being a good web citizen so it meets all of our requirements to be used in a URI Template Answer Generator.


Creating the Answer Generator

Now let's create the Answer Generator. It's just a matter of filling in fields correctly. But before we do that, let's see if Yummly supports HTTPS, which will protected users' search queries from being snooped. If you replace the http in the Yummly URL with https, and load it in your browser, it still works. Great! So let's use https for all URLs pointing to Yummly. Our target URL will be:

https://www.yummly.com/recipes?q=hummus&noUserSettings=true&allowedIngredient=&excludedIngredient=&maxTotalTimeInSeconds=Any+time&flavor.salty=Dislike&flavor.savory=+&flavor.sour=+&flavor.bitter=Really+Dislike&flavor.sweet=+&flavor.spicy=Like&allowedDiet=387%5ELacto-ovo+vegetarian&allowedAllergy=398%5ESeafood-Free&nutrition.cholesterol=+&nutrition.fat=+&nutrition.calories=+&nutrition.carbs=70+g%2Fserving&imagesOnly=false&blogsOnly=false&myRecipesOnly=false&sortBy=relevance

Now go to the Plugins page. If it's not selected already, select the Answer Generators tab. At the bottom of the table, select the button Create an Answer Generator. You should be taken to a long form which looks quite intimidating. Don't worry, we're going to fill it in step by step below.

Name: Yummly Test 1

This is whatever you want to use as the name of the Answer Generator, and will be shown to users when they select Answer Generators to add. Fill in Yummly Test 1. The reason for the Test 1 suffix is that Answer Generators must have unique names. Maybe somebody will already use this name so it's best to substitute a random number for 1.

Icon URL: https://www.yummly.com/favicon.ico

This optional field lets you set a small icon to be displayed next to the label for answers. The first URL to try is made by replacing the path in the target URL with /favicon.ico. So would be https://www.yummly.com/favicon.ico. If you load that URL in your browser, you'll see a little y icon. This is exactly what we need so we'll use it for the value of the Icon URL.

What if that guessed icon URL didn't work? We can go to the target URL and view the source of the page. In the source, do a search for icon. You'll see lines like this:

<link rel="shortcut icon"
 href="https://yummly-static.a.ssl.fastly.net/yummly-website/a46e8b0d395e4871b0606350b0c451b12f4ce9f0/favicon.ico" sizes="16x16 32x32 64x64">
<link rel="icon"
 href="https://yummly-static.a.ssl.fastly.net/yummly-website/a46e8b0d395e4871b0606350b0c451b12f4ce9f0/img/favicon.png" type="image/png">

Either of those href values would be fine, although we prefer smaller files, so most likely the first one would be better. If you can't find a suitable URL, it's fine to leave this field blank. Also https URLs are preferred over http URLs which cause browser warnings when loaded in solveforall.com, since solveforall.com uses https.

Removed Query Terms: recipe,recipes,yummly,for

These are terms that will be removed from the user's query before they are passed to the Answer Generator. We want to transform phrases like recipes for tacos, tacos recipe, yummly tacos to tacos, so we want to ignore the terms recipe,recipe, yummly, and for. To do that, we'll concatenate all these terms together, separating terms with a comma. So we'll fill in recipe,recipes,yummly,for.

Removed Query Term Boosts: 1,0.5,0.5,0

If the user uses the terms recipe,recipe, or yummly, it's very likely a link to Yummly is relevant. So we'll boost the relevance by 0.5 if recipe or recipe are found, ensuring the Yummly link is included in the results. We'll boost yummly itself to ensure that if yummly appears in the query, the Yummly link is the top result (above other recipe links). The presence of the for term shouldn't affect the relevance, so we'll set the boost to 0. Now we take the value of the Removed Query Terms, recipe,recipes,yummly,for, and for each term, substitute the desired boost, to get 1,0.5,0.5,0.

Provider: Yummly

Hit the Set Provider button and use the search form to find find Yummly which the Content Provider of the pages we are linking to. Check the checkbox for Yummly, and hit the Set Categories button.

If you can't find the Content Provider of your Answer Generator, you could either create a Content Provider starting from the Plugins page, or just leave field unset. The advantage of associating a Content Provider is that users will be able to find your plugin when browsing Content Providers in the customization section of Solve for All.

Required Content Recognizers: (Leave unset)

We don't need to detect any patterns in the query, so no content recognition is needed and we'll leave this unset.

Required Semantic Data Collections: (Leave unset)

We don't need to search any Semantic Data Collections for this plugin to work, so we'll leave this unset. In the future, possibly we could upload a list of ingredients or dish names that could be searched, and only output if an ingredient or dish is found, but that isn't necessary.

Categories: Food, Gastronomy

Hit the Set Categories button. A popup should appear letting you choose one or more categories. The categories that make sense for a recipe finder are Food and Gastronomy, so we'll pick those. Search for them using the search form in the top right corner, and check them. When done, hit the Set Categories button.

Type: URI Template

Since we're outputting a link from a template without code, we'll set the type of Answer Generator to URI Template.

Recognition Keys: (Leave blank)

Our first pass at this Answer Generator will just pass the entire query (minus the removed terms) to the Answer Generator without trying to recognize anything. Therefore, we don't need recognition keys, and we'll leave this blank.

Label: Yummly Recipe Search

This is the label of the link that will be generated. Yummly Recipe Search seems appropriate, so let' go with that.

Tooltip: Search for recipes on Yummly

This is the tooltip of the link that will be generated. When a user hovers over the link with a mouse, this text will be displayed. Let' go with Search for recipes on Yummly.

Summary: (Leave blank)

We want the link that this Answer Generator outputs to be a Quick Link, so that it goes to the top of a column. Quick links can't have any summaries so we'll leave this blank.

URI Prefix: https://www.yummly.com/recipes?q=

Now we're getting to the heart of the Answer Generator. The URI prefix is the part of the URL that is before the query. In our target URL, we copy the part before hummus, which is https://www.yummly.com/recipes?q=.

URI Middle: (Leave blank)

If we leave this blank, either the entire query will be substituted here, or if we were looking for recognition results, each substring that was recognized. This default makes sense whether we recognize anything or not, so we'll leave it blank.

URI Suffix: &noUserSettings=true&allowedIngredient=&excludedIngredient=&maxTotalTimeInSeconds=Any+time&flavor.salty=Dislike&flavor.savory=+&flavor.sour=+&flavor.bitter=Really+Dislike&flavor.sweet=+&flavor.spicy=Like&allowedDiet=387%5ELacto-ovo+vegetarian&allowedAllergy=398%5ESeafood-Free&nutrition.cholesterol=+&nutrition.fat=+&nutrition.calories=+&nutrition.carbs=70+g%2Fserving&imagesOnly=false&blogsOnly=false&myRecipesOnly=false&sortBy=relevance

This is the part after the query in the target URL. For now, it includes my dietary preferences and restrictions, but later we'll come back to make this Answer Generator honor each user's preferences.

Substitutions

(Leave blank)

No workarounds for special characters are needed, so we'll leave this blank.

Relevance: (Leave blank)

If this is left blank, the answer produced by this Answer Generator will have a relevance equal to the recognition level of recognized content, or 0 if nothing was recognized. We don't care that much about the relevance, since the link will be at the top in the quick links section, so we'll leave this blank. The Removed Query Term Boosts will help set the final relevance based on terms found in the query.

Max Results: 1

We only want a single link output by this Answer Generator, so we'll leave this at 1.

Embeddable: (Check)

If this is checked, the user will be able to preview the page in an iframe, without leaving Solve for All. But most sites don't allow being embedded in an iframe on another site. We'll check this for now, but we'll have to make sure embedding works.

Should Embed: (Uncheck)

Checking this property embeds the link as an iframe to display Since the Yummly search results page takes a fair amount of space, we'll leave this unchecked because we don't want to show small portions of Yummly pages on the answers page.

Minimum Width, Preferred Width, Minimum Height, Preferred Height: (Leave blank)

These properties only apply when Should Embed is checked, so we' leave them blank.

Requires Recognition Results: (Leave checked)

Unchecking this box could theoretically speed up searches because the Answer Generator could execute before all triggers are completed. But since the Answer Generator is just a URI Template, it's relatively fast to execute anyway. Leaving this box check will also let us use recognition results later. So we'll leave this checked.

Default Activation Code: yummlyt1

We can pick a default activation code that will let any user activate this Answer Generator even if she has not installed it yet. Activation codes have to be unique, so if another Answer Generator has the same activation code, we'll get an error when we submit the form. If you'll following along this guide, you'll probably want to use a different suffix to avoid uniqueness errors.

Also, keep in mind that activation codes less than 5 characters have to be approved by Solve for All before they become active.

Required Permissions, Optional Permissions: (Leave blank)

We don't need any special permissions like reading the user's location, so we'll leave these blank. Having required permissions means that the user will have to grant permissions before the Answer Generator is executed, leading to more friction.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Answer Generator will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Associate with Any Trigger: (Check)

Since we could theoretically search Yummly for anything, we want to associate this Answer Generator with the Any Trigger. That way, the user will be given a choice to activate the Answer Generator with the Any Trigger.

If This Answer Generator is included in an Engine (say a recipe Engine), and the Engine is explicitly activated, a link to Yummy will be produced as long as the user chooses to activate this Answer Generator with the Any Trigger

Default Descriptor - Name: (Leave blank)

These Default Descriptor properties are fallbacks for localized properties in case the specialized properties don't exist for the user's locale. By default, the name of the Default Descriptor is filled in with the Name property above, so we can leave this blank.

Default Descriptor - Description: Search for recipes on Yummly

Fill this field in with a good description of this Answer Generator; we'll use Search for recipes on Yummly.

Default Descriptor - Catalog Small Image: (Ignore)

If we had a good square image representing Yummly (about 128x128), we could choose it here, but since we don't, we'll ignore this field.

Save

Now that we are done filling in all the fields, we'll hit the Save button. Now the Answer Generator is saved and ready to go!

Testing the Answer Generator

Let's test it out be activating the Answer Generator with its activation code. Assuming you picked yummlyt1 as the activation code, try searching for ?yummlytl pita. If you get the correct link, try click on the expand icon. Looks like Yummly can be embedded in an iframe, so we can leave the Embeddable field checked. If the linked page doesn't show, we should uncheck the Embeddable checkbox. To fix this or make other changes, find your newly saved Answer Generator in the Plugins page, in the Answer Generators tab. You can enter the name of your Answer Generator in the search filter to find the Answer Generator you just created, then click on the name. You'll be taken to a page that has an Edit button on it. Click that and fix the fields as necessary.

At this point, any user can choose to activate the Answer Generator either with the Any Trigger, the activation code ?yummlyt1, or by choosing customized keywords (defaulting to recipe, recipes, and yummly). To see this, let's find the Answer Generator in the customization section. Go to the Find Plugins page and search for your Answer Generator. Then go to its detail page, and in the Your Configuration section, hit the Add button. On the first step, select an Engine that is active by default. On the next step, notice that recipe, recipes, and yummly are the default activation keywords. Try using the default set of activation keywords, and saving this configuration. Then search for veggie burger recipe. You should get a quick link to Yummly. (If not, check that the Engine you selected was active by default.) Congratulations, you've just created an Answer Generator for anyone to use!


(Optional) Creating a Trigger

It would be nice if this Answer Generator could produce a Yummly link whenever there is mention of food in the answery query. To see if this is possible, let's see what happens when we search for a food; try searching for ?debug tacos. The Debug Answer Generator shows the recognition results which look like this:

    {
      "org.dbpedia.ontology.Food": [
        {
          "recognitionLevel": 0.3658862,
          "localRecognitionLevel": 0.45735776,
          "humanFormattedWikipediaArticle": "Taco",
          "wikipediaArticleName": "Taco",
          "matchedText": "Taco"
        }
      ],
      ...
      "eu.fbk.wndomains.Gastronomy": [
        {
          "recognitionLevel": 4.0718384,
          "localRecognitionLevel": 4.3783207,
          "matchedText": "tacos"
        },
        {
          "recognitionLevel": 2,
          "localRecognitionLevel": 2,
          "matchedText": "tacos"
        }
      ]
    }
  

So it looks like the Knowledge Base and WordNet domain categorization can recognize foods with the recognition keys org.dbpedia.ontology.Food and eu.fbk.wndomains.Gastronomy, respectively.

Now let's look at what Triggers are already available in the Plugins page. If you do a search for Feed or Gastronomy, you'll find existing Triggers there. At this point we would probably just use those, but for the sake of the guide, let's create our own custom Trigger. The Trigger will activate any Answer Generators associated with it if the the recognition result mapped to the recognition key org.dbpedia.ontology.Food has a recognition level of 0.35 or more, which barely allows the query tacos to pass. To implement this rule, we won't have to write any code, we just need to create a simple Trigger.

Go to the Plugins page and select the Triggers tab. Then hit the Create a Trigger button after the table. You'll be taken to a form to create a trigger. Let's start filling it in.

Name: Yummly Test 1

This is whatever you want to use as the name of the Trigger, and will be shown to users when they select Triggers to activate their Answer Generators. Fill in Yummly Test 1. The reason for the Test 1 suffix is that Triggers must have unique names. Maybe somebody will already use this name so it's best to substitute a random number for 1.

Required Content Recognizers: (Leave as is)

Again, we don't need to detect any patterns in the query, so no content recognition is needed and we'll leave this unset.

Required Semantic Data Collections: (Leave as is)

Again, we don't need to search any Semantic Data Collections for this plugin to work, so we'll leave this unset.

Documentation file: (Leave unset)

Once we are sure the Trigger works as we expect, we could upload a file describing how it works here, but for now we can leave this unset.

Type: Simple

Simple Triggers implement the simple rule: if the recognition level mapped to a recognition key is greater than some threshold, activate the Answer Generator. This is good enough and doesn't require us to write any code, so let's pick Simple.

Recognition Keys: org.dbpedia.ontology.Food

This is the recognition key corresponding to a food classified by DBpedia. We could add the recognition key eu.fbk.wndomains.Gastronomy, but recognition levels associated with WordNet domains tend to be a little high and may not actually signify that a word belons to the domain. So we'll just use org.dbpedia.ontology.Food.

Min Recognition Level: 0.3

Since tacos was recognized with the key org.dbpedia.ontology.Food at a recognition level of around 0.36, we'll set the recognition level to below this level to ensure tacos matches. If we get a lot of false positives or false negatives, we can come back and adjust this level later. For now, let's use 0.3 which is a bit lower than 0.36.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Trigger will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Default Descriptor - Name: (Leave blank)

By default, the name of the Default Descriptor is filled in with the Name property above, so we can leave this blank.

Default Descriptor - Description: A food matched by DBpedia with recognition level 0.3 or more

Fill this field in with a good description of this Trigger; we'll use A food matched by DBpedia with recognition level 0.3 or more.

Save

Now we are done filling in all the fields, we'll hit the Save button. Now the Trigger is saved and ready to go! That was easy.


(Required if you created a Trigger) Creating an Answer Generator - Trigger Association

Before the user can use this Trigger with the Answer Generator we created, we need to tell Solve for All that they can be used together. To do that, we'll need to create an Answer Generator - Trigger Association. Don't worry -- it's short and sweet. Go to the Plugins page and click the Associations tab. Find the button Create an Association and click it. You will be taken to a short form to create an Association. Let's fill it in.

Answer Generator: Yummly Test 1

Hit the Set Answer Generator button and find the Answer Generator you just created. When done, hit the Set Answer Generator button.

Trigger: Yummly Test 1

Hit the Set Trigger button and find the Trigger you just created. When done, hit the Set Trigger button.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Association will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Save

Now we are done filling in all the fields, we'll hit the Save button. Now the Association is available to all users. That was even easier than creating a Trigger.

Try it out

Let's add this association and try it out. Find the Answer Generator in the customization section again. Go to the Find Plugins page and search for your Answer Generator. Then go to its detail page, and in the Your Configuration section, hit the Add button. First select an Engine that is active by default. On the next step, you can configure how you want to activate this Answer Generator. Notice that your new Trigger is listed. Select it, save your settings, and do a search for tacos. You should see the Yummly link show up. If not, check that the Engine you selected was active by default.

If everything is working, you have successfully associated your Answer Generator with an activation Trigger that makes sense and all other users can do the same. If you share the Engine that contains your configured Answer Generator, any user that adds your Engine will automatically get this configuration.


(Optional) Creating a Preferences Page

Coming soon! For now, check out the documentation for User Preferences Pages.

Next Steps

Now take this example and tweak it for you own use case. We're looking forward to see what awesome stuff you come up with!

For advanced usage, you can check the documentation of Answer Generators for more details.