Guide - Creating a Content Template Answer Generator

Overview

In this guide, we're going to show you how we created the Font Awesome Answer Generator, which produces inline blocks of content. (Font Awesome is a free font that can be used to show vector icons in HTML pages.) We'll also create the Semantic Data Collection that provides the searchable data for it.

Why do this in first place? Why not just let the user go to Font Awesome and find the icons?

Here's a few reasons:
  • Navigating to another site requires the user to find the right source and click on it. Opening another site is especially painful on mobile devices because most devices require another window to be opened. If the user just wants a small bit of information, this is too much overhead.
  • The target site might not be optimized for mobile devices, or you might want show a custom presentation of the data other than what is on the web site.
  • By adding Font Awesome documentation to Solve for All, the user will be able to auto-complete the names of Font Awesome icons in Solve for All's search input box.

Of course, Font Awesome documentation is just an example. There are many sources of data that might not even be scrapeable. You can apply this guide to making Answer Generators for other sources of data.


Preliminaries

Before we dive in, let's make sure we aren't wasting our time and that what we want to do is feasible. First let's check if an Answer Generator for Font Awesome is available. Go to Solve for All's plugin search page and enter "font awesome". Select Font Awesome Icons, and you will be taken to its detail page. In the detail page, go to the Try it out section and enter fa-pied-piper. You should be taken to the Solve for All answers page that shows the Pied Piper icon. At this point, you would probably just add this Answer Generator instead of creating your own (unless this Answer Generator didn't do everything that you wanted). But this Answer Generator was the result of creating this guide, so let's continue!

Finally, let's see if it's possible to scrape the data from Font Awesome. Go to the cheatsheet. Looks like all the icons are there. If you inspect an icon using Chrome web tools, it looks like the data for each icon is an element in the CSS selector .container .row .fa.fa-fw. It looks like web scraping will work fine so we can continue making this Answer Generator.


Creating the Semantic Data Collection

Extracting the data

First, let's scrape the cheatsheet to create a Semantic Data Collection that will supply the data for the Answer Generator. Assuming you have ruby installed on your system, clone the repository to scrape the icons:

git clone https://github.com/jtsay362/font-awesome-populator

Before we run the extractor, let's briefly cover what the code is doing. In populate.rb:

  1. First, the program downloads the cheatsheet locally so it doesn't have to be downloaded every time the program is run.
  2. The objective of the program is to create a valid Semantic Data Changefile which is described in the documentation for Semantic Data Collections.
  3. After opening the output file that will be the changefile, the program writes the metadata for the changefile:
    {
      "metadata" : {
        "settings" : {
          "analysis": {
            "char_filter" : {
              "no_special" : {
                "type" : "mapping",
                "mappings" : ["-=>", "f=>", "a=>"]
              }
            },
            "analyzer" : {
              "lower_whitespace" : {
                "type" : "custom",
                "tokenizer": "whitespace",
                "filter" : ["lowercase"],
                "char_filter" : ["no_special"]
              }
            }
          }
        },
        "mapping" : {
          "_all" : {
            "enabled" : false
          },
          "properties" : {
            "name" : {
              "type" : "string",
              "analyzer" : "lower_whitespace"
            },
            "suggest" : {
              "type" : "completion",
              "analyzer" : "lower_whitespace"
            }
          }
        }
      },
      ...
          
    Notes:
    • It uses a special analyzer that ignores the f, a, and - characters. This allows it to treat the query fa-nav-icon the same as navicon, for the purpose of searching and auto-completion.
    • It disables the _all mapping which saves space
    • It maps the suggest field to the completion type which enables auto-complete
    • For more information on mapping, see the Elasticsearch documentation
  4. Then it starts outputting the updates section of the changefile, with one document per icon. It uses nokogiri to parse the document and select elements with the selector .container .row .fa.fa-fw. For each matching element, it extracts the icon's CSS class from CSS classes of the element. The CSS class is used for the name and the suggest fields. So it will append something like this:
      "updates" :
      [
        {
          "name" : "fa-500px",
          "suggest" : "fa-500px"
        },
        {
          "name" : "fa-adjust",
          "suggest" : "fa-adjust"
        },
        ...
      ]
    }
  5. Finally, it bzip's the changefile up so it can be uploaded to Solve for All.

Now let's run the populator:

cd font-awesome-populator
bundle install
ruby populate.rb -d

The file font-awesome-icons.json.bz2 should now exist. Now that we've done the hard part, let's upload this to Solve for All. Go to the Plugins page. If it's not selected already, select the Semantic Data Collections tab. At the bottom of the table, select the button Create a Semantic Data Collection. You should be taken to a long form which looks quite intimidating. Don't worry, we're going to fill it in step by step below.

Name: Font Awesome Icons Test 1

This is whatever you want to use as the name of the Semantic Data Collection, and will be shown to developers that want to search your collection. Fill in Font Awesome Test 1. The reason for the Test 1 suffix is that Semantic Data Collections must have unique names. Maybe somebody will already use this name so it's best to substitute a random number for 1.

Query Template: {"match" : {"name" : "{{query}}"}}

The query template is a Mustache template that resolves to the actual query that will be sent to Elasticsearch. {{query}} will be replaced with the query that the user types, so if she searches for cogs, the template will evaluate to:

{
  "match" : {
    "name" : "cogs"
  }
}

This is the simplest possible query that just tries to match the query with the name. For more advanced searching, see the Elasticsearch Query DSL.

Default Recognition Key: com.example.recognition.web.FontAwesomeIcon

You can choose any recognition key to group results from this Semantic Data Collection with. You may want to check the existing recognition keys for a conflict, but since Solve for All is just starting out, you're not likely to be in conflict with any other key. So we'll choose com.example.recognition.web.FontAwesomeIcon which seems descriptive enough.

We had to specify a Default Recognition Key because each document omitted the recognitionKeys property. So the default recognition key will be used for any matched document.

Boost factor: 1

If we find that our search queries result in low or high recognition levels, we can apply a multiplicative factor to the recognition level. Since we don't know what the recognition levels will be yet, we'll leave this as 1.

Removed Query Terms: fa,font,awesome,fontawesome

These are terms that will be removed from the user's query before they end up in query variable. We want to transform phrases like fa fa-pause-circle, font awesome fa-pause-circle, and fontawesome fa-pause-circle to fa-pause-circle, so we want to ignore the terms fa,font, awesome, and fontawesome. To do that, we'll concatenate all these terms together, separating terms with a comma. So we'll fill in fa,font,awesome,fontawesome.

Removed Query Term Boosts: (leave blank)

We will boost the relevance based on keywords in the Answer Generator, so there's no need to apply a boost here. So we'll leave this blank.

Name Fields(s): (leave blank)

Solve for All adjusts the Elasticsearch-computed score of a matched document by adding a score of the similarity between the answer query and the value of a name field. This way exact matches get an extra boost. By default, Solve for All uses the field name which is where we put the icon names in the Semantic Data Collection. So we can just leave this blank. If we used a different field for the names of each document, we would use that field name here.

Suggestion Icon URL: https://fortawesome.github.io/Font-Awesome/assets/ico/favicon.ico

When auto-completing a query, Solve for All displays icons to indicate which Semantic Data Collection had a matching result. We want an icon that represents Font Awesome. To find one, we'll look at the source of the main page of Font Awesome. It contains the following line:

<link rel="shortcut icon" href="./assets/ico/favicon.ico">

and if we click the href attribute, we get taken to the URL

https://fortawesome.github.io/Font-Awesome/assets/ico/favicon.ico

which is a nice icon. So we'll use this URL.

If you can't find an icon that represents your collection, it's ok; the icon is optional. Also note that https URLs are preferred over http URLs which cause browser warnings when loaded in solveforall.com, since solveforall.com uses https.

Documentation file: (Leave unset)

Once we are sure the Semantic Data Collection works as we expect, we could upload a file describing the document contents, but for now we can leave this unset.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Semantic Data Collection will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Save

Now that we are done filling in all the fields, we'll hit the Save button. Now the Semantic Data Collection is ready to accept our data.

Upload the changefile

We are ready to upload the changefile our Ruby program created. Find the Semantic Data Collection in the Plugins page. Select the Semantic Data Collections if it isn't selected yet, and search for font awesome. You should find your newly created Semantic Data Collection. Select it and you'll be taken to its detail page. At the bottom of the detail page, there is a button labeled Upload a file. Click that and select the file font-awesome-icons.json.bz2. (It doesn't matter whether or not the checkbox labeled Remove existing data is checked, since this is your first upload. If you have to upload again, you should check this box.)

If your changefile was valid, you should get a success message. Otherwise, check that your changefile was valid JSON. Once you the upload completes successfully, this Semantic Data Collection is now ready to be searched!


Creating an Content Template Answer Generator

Let's create a content template that will show results from this Semantic Data Collection in an inline answer. You can create one in a new project directory, but if you use the example plugins project, you'll gain the ability to validate your template and test your Answer Generator. So let's clone the repo and create a directory for our template

git clone https://github.com/jtsay362/solveforall-example-plugins
cd solveforall-example-plugins
mkdir src/answer_generators/font_awesome_test

In solveforall-example-plugins/src/answer_generators/font_awesome_test, create a new file font_awesome_test.html.ejs with the following contents:

<%
let name = null;
try {
  const recognitionResult = recognitionResults['com.example.recognition.web.FontAwesomeIcon'][0];
  name = recognitionResult.name;
} catch (e) {
  throw e;
}

const url = 'https://fortawesome.github.io/Font-Awesome/icon/' + name.substring(3) + '/';
%>
<!DOCTYPE html>
<html>
<head>
  <title>Font Awesome</title>
  <meta name="com.solveforall.meta.answer.uri" content="<%= url %>" >
</head>

<body>
  <div style="margin-left: 0.5em; font-size: 48px; line-height: 1.5em;">
    <i class="fa <%= name %>"></i>
    <%= name %>
  </div>
</body>
</html>

This is an EJS template, pretty similar to ERB, JSP, or ASP if you're familiar with those. Things to note in the template:

  • <% ... %> are code blocks
  • <%= ... %> blocks output the inner Javascript expression, which is HTML escaped
  • recognitionResult is computed by extracting the first result in the list mapped to the recognition key we chose
  • Since we're going to make this content server-side sanitized, the content will inherit the Bootstrap 3.3.6 Spacelab styles and also have access to Font Awesome 4.5 icons. So we can go ahead and show the Font Awesome icon the normal way, with <i class="fa fa-bars"></i>, for example.

If you have Apache Ant and npm installed, you can check that this is valid EJS. In the project root directory, enter the following commands:

# Only needs to be run once
npm install
ant

If ant succeeds without reporting errors, that means the .ejs file is valid. To upload it to Solve for All, go to the Plugins page. If it's not selected already, select the Answer Generators tab. At the bottom of the table, select the button Create an Answer Generator. You should be taken to a long form which looks quite intimidating. Don't worry, we're going to fill it in step by step below.

Name: Font Awesome Test 1

This is whatever you want to use as the name of the Answer Generator, and will be shown to users when they select Answer Generators to add. Fill in Font Awesome Test 1. The reason for the Test 1 suffix is that Answer Generators must have unique names. Maybe somebody will already use this name so it's best to substitute a random number for 1.

Icon URL: https://fortawesome.github.io/Font-Awesome/assets/ico/favicon.ico

Since we found this icon before, let's use it again.

Removed Query Terms: fa,font,awesome,fontawesome

The same terms as in the Semantic Data Collection. We don't really need to modify the query, since we never use the variable q in our content template, but we want to boost the relevance if these terms are found in the answer query.

Removed Query Term Boosts: 1,0.2,0.2,1

If the user uses the terms fa,font, awesome, or fontawesome, it's very likely the Font Awesome icon is what the user wants. So we'll boost the relevance by 0.2 if font or awesome are found, helping the icon go to the top of the results. We'll boost fa and fontawesome to ensure that if these terms appear in the query and a matching icon is found, the answer we produce is the top result. Now we take the value of the Removed Query Terms, fa,font,awesome,fontawesome, and for each term, substitute the desired boost, to get 1,0.2,0.2,1.

Provider: (Leave unset)

This is optional, and there's probably not a lot of other answers that Fort Awesome (the maker of Font Awesome) can produce, so we'll leave this unset.

Required Content Recognizers: (Leave unset)

We don't need to detect any patterns in the query (since we're matching by icon name in the Semantic Data Collection), so no content recognition is needed and we'll leave this unset.

Searched Semantic Data Collections: Font Awesome Icons Test 1

We want to search our Semantic Data Collection whenever this Answer Generator is active, so we'll pick it from the table of available Semantic Data Collections that pops up when you hit the Set Searched Semantic Data Collections button.

Categories: Programming

Hit the Set Categories button. A popup should appear letting you choose one or more categories. The category that makes sense for icons to be put on web pages is Programming, so we'll pick that. Search for it using the search form in the top right corner, and check them. When done, hit the Set Categories button.

Type: Content Template

Since we're outputting inline content from a template, we'll set the type of Answer Generator to Content Template.

Recognition Keys: com.example.recognition.web.FontAwesomeIcon

We'll use the same recognition key we set as the default recognition key in the Semantic Data Collection. If something in the query matches the name of an icon, this key will be mapped to a list of documents, and this Answer Generator will be eligible to run.

Label: Font Awesome Icon

This is the label of the link that will be generated. Font Awesome Icon seems appropriate, so let' go with that.

Tooltip: (Leave blank)

A tooltip for the answer isn't needed, and may be distracting, so let's leave the tooltip blank.

Relevance: (Leave blank)

If this is left blank, the answer produced by this Answer Generator will have a relevance equal to the recognition level of a matching document, plus the Removed Query Term Boosts if a Removed Query Term is found. This is what we want, so we'll leave the relevance blank.

Max Results: 1

We only want a single inline block from this Answer Generator, so we'll leave this at 1.

Content Template File: (Select font_awesome_test.html.ejs)

Attach the EJS file we just created.

Server Side Sanitized: (Check)

We don't need to run Javascript in the inline answers the Answer Generator produce, and we want to inherit the styles and icons of the answer page, so we'll check this.

Minimum Width, Preferred Width, Minimum Height, Preferred Height: (Leave blank)

Since the content will be server-side sanitized, the dimensions of the inline answer will automatically fit the contents. So we can leave these all blank.

Requires Recognition Results: (Leave checked)

Since we need the results of searching the Semantic Data Collection before this Answer Generator runs, we need to leave this checked.

Default Activation Code: fatest1

We can pick a default activation code that will let any user activate this Answer Generator even if she has not installed it yet. Activation codes have to be unique, so if another Answer Generator has the same activation code, we'll get an error when we submit the form. If you'll following along this guide, you'll probably want to use a different suffix to avoid uniqueness errors.

Also, keep in mind that activation codes less than 5 characters have to be approved by Solve for All before they become active.

Required Permissions, Optional Permissions: (Leave blank)

We don't need any special permissions like reading the user's location, so we'll leave these blank. Having required permissions means that the user will have to grant permissions before the Answer Generator is executed, leading to more friction.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Answer Generator will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Associate with Any Trigger: (Leave unchecked)

We don't want to allow users to trigger this Answer Generator unless an icon name is found in the query. So we will leave this unchecked.

Default Descriptor - Name: (Leave blank)

These Default Descriptor properties are fallbacks for localized properties in case the specialized properties don't exist for the user's locale. By default, the name of the Default Descriptor is filled in with the Name property above, so we can leave this blank.

Default Descriptor - Description: Show Font Awesome icons

Fill this field in with a good description of this Answer Generator; we'll use Show Font Awesome icons.

Default Descriptor - Catalog Small Image: (Downloaded copy of https://fortawesome.github.io/Font-Awesome/assets/ico/favicon.ico)

The favicon for Font Awesome is quite big, so we'll download it and attach it here.

Save

Now that we are done filling in all the fields, we'll hit the Save button. Now the Answer Generator is saved and ready to go!

Testing the Answer Generator

Let's test it out be activating the Answer Generator with its activation code. Assuming you picked fatest1 as the activation code, try searching for ?fatest1 bolt. You should see the bolt icon in a content block. Nice!

At this point, any user can choose to activate the Answer Generator either with the activation code ?fatest1 or by choosing customized keywords (defaulting to fa, fontawesome, font, and awesome). To see this, let's find the Answer Generator in the customization section. Go to the Find Plugins page and search for your Answer Generator. Then go to its detail page, and in the Your Configuration section, hit the Add button. On the first step, select an Engine that is active by default. On the next step, notice that fa, fontawesome, font, and awesome are the default activation keywords. Try using the default set of activation keywords, and saving this configuration. Then search for fa camera. You should get a block of content with the camera icon. (If not, check that the Engine you selected was active by default.) Congratulations, you've just created an Answer Generator for anyone to use!


(Optional) Creating a Trigger

It would be nice if this Answer Generator could produce a block of content whenever the name of a Font Awesome icon is in the query. This would be for web developers that use Font Awesome often. To do this, let's see what happens when we search for a food; try searching for ?debug ?fatest1 comments. You should see both a Debug inline answer and an inline answer containing the comments icon. The Debug Answer Generator shows the recognition results which look like this:

{
  "com.example.recognition.web.FontAwesomeIcon": [
    {
      "name": "fa-comments",
      "recognitionLevel": 0.6467845,
      "localRecognitionLevel": 0.6467845,
      ...
    }
  ],
  ...
}

So it looks like matches of icon names will have a recognition level at a little more than 0.6. So let's create a Trigger that implements the rule,

If there is a recognition result mapped to the key
com.example.recognition.web.FontAwesomeIcon, and its recognition
level is at least 0.6, activate the Answer Generator.

To do this, we need to create a Simple Trigger. Go to the Plugins page and select the Triggers tab. Then hit the Create a Trigger button after the table. You'll be taken to a form to create a trigger. Let's start filling it in.

Name: Font Awesome Icon Test 1

This is whatever you want to use as the name of the Trigger, and will be shown to users when they select Triggers to activate their Answer Generators. Fill in Font Awesome Test 1. The reason for the Test 1 suffix is that Triggers must have unique names. Maybe somebody will already use this name so it's best to substitute a random number for 1.

Required Content Recognizers: (Leave as is)

Again, we don't need to detect any patterns in the query, so no content recognition is needed and we'll leave this unset.

Required Semantic Data Collections: Font Awesome Icons Test 1

This Trigger depends on the result of searching the Semantic Data Collection we created to see if there are results mapped to the com.example.recognition.web.FontAwesomeIcon recognition key, so we'll pick it from the table of available Semantic Data Collections that pops up when you hit the Set Searched Semantic Data Collections button.

Documentation file: (Leave unset)

Once we are sure the Trigger works as we expect, we could upload a file describing how it works here, but for now we can leave this unset.

Type: Simple

Simple Triggers implement the simple rule: if the recognition level mapped to a recognition key is greater than some threshold, activate the Answer Generator. That's exactly what we want, so let's pick Simple.

Recognition Keys: com.example.recognition.web.FontAwesomeIcon

This is the same default recognition key of the Semantic Data Collection we created.

Min Recognition Level: 0.6

Since comments was recognized with the key com.example.recognition.web.FontAwesomeIcon at a recognition level of around 0.64, we'll set the recognition level to below this level to ensure comments matches. If we get a lot of false positives or false negatives, we can come back and adjust this level later. For now, let's use 0.6 which is a bit lower than 0.64.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Trigger will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Default Descriptor - Name: (Leave blank)

By default, the name of the Default Descriptor is filled in with the Name property above, so we can leave this blank.

Default Descriptor - Description: Activates when the name of a Font Awesome icon is found

Fill this field in with a good description of this Trigger; we'll use Activates when the name of a Font Awesome icon is found.

Save

Now we are done filling in all the fields, we'll hit the Save button. Now the Trigger is saved and ready to go! That was easy.


(Required if you created a Trigger) Creating an Answer Generator - Trigger Association

Before the user can use this Trigger with the Answer Generator we created, we need to tell Solve for All that they can be used together. To do that, we'll need to create an Answer Generator - Trigger Association. Don't worry -- it's short and sweet. Go to the Plugins page and click the Associations tab. Find the button Create an Association and click it. You will be taken to a short form to create an Association. Let's fill it in.

Answer Generator: Font Awesome Icon Test 1

Hit the Set Answer Generator button and find the Answer Generator you just created. When done, hit the Set Answer Generator button.

Trigger: Font Awesome Icon Test 1

Hit the Set Trigger button and find the Trigger you just created. When done, hit the Set Trigger button.

Publicly Visible, Publicly Linkable: (Check)

If these are checked, this Association will be usable by all users. Otherwise, only you can use it. Since we want to share our knowledge, we'll leave these checked.

Save

Now we are done filling in all the fields, we'll hit the Save button. Now the Association is available to all users. That was even easier than creating a Trigger.

Try it out

Let's add this association and try it out. Find the Answer Generator in the customization section again. Go to the Find Plugins page and search for your Answer Generator. Then go to its detail page, and in the Your Configuration section, hit the Add button. First select an Engine that is active by default. On the next step, you can configure how you want to activate this Answer Generator. Notice that your new Trigger is listed. Select it, save your settings, and do a search for comments. You should see the Font Awesome inline answer show up. If not, check that the Engine you selected was active by default.

If everything is working, you have successfully associated your Answer Generator with an activation Trigger that makes sense and all other users can do the same. If you share the Engine that contains your configured Answer Generator, any user that adds your Engine will automatically get this configuration.


Next Steps

Now take this example and tweak it for you own use case. We're looking forward to see what awesome stuff you come up with!

For advanced usage, you can check the documentation of Semantic Data Collections, Answer Generators, and Inline Answers for more details.