The api/service/mqlread with extended=true API implements the Extended MQL service.
Extended MQL augments MQL by allowing properties to call out to external data sources or to run arbitrary code to dynamically compute results for a MQL query.
See the gallery of documented extension properties for examples.
To invoke extended MQL, use the new "extended" api/service/mqlread envelope parameter and set it to true.
Extension properties may be defined in one of two namespaces: either on a type, like regular MQL properties, or in the /freebase/emql namespace which defines a global namespace specific to extended MQL.
Extension properties defined on a type are addressed using the same syntax as regular MQL properties whereas global extended MQL properties - defined within the /freebase/emql namespace - are addressed by prefixing them with the '@' character.
Use the query editor to edit and run one of the MQL queries below.
Query to create an extension property on a type:
{
"create": "unless_exists",
"name": <name of property>
"id": null,
"/type/property/schema": {
"id": <id of type to host property>
},
"key": {
"value": <name of property>,
"namespace": <id of type to host property>
},
"/type/extension/uri": {
"value": <uri of adapter to invoke>
},
"/type/extension/pre": {
"value": true
},
"/type/extension/fetch": {
"value": true
},
"/type/extension/reduce": {
"value": false
},
"/type/extension/help": {
"value": false
},
"type": [
"/type/property",
"/type/extension"
],
"/type/extension/api_key_names": [
"zero", "or", "more", "api", "key", "names", ...
],
}
The values for /type/extension/pre, /type/extension/fetch, /type/extension/reduce and /type/extension/help are set to their default values and can be changed to suit the property's needs with a one of the queries below.
Update a phase property on an extension property:
{
"type": "/type/extension",
"id": <id of property to update>,
"/type/extension/<phase>": {
"value": <true or false>,
"connect": "update"
}
}
While Metaweb's Acre service is ideally suited for implementing an HTTP adapter in javascript, any server answering HTTP requests over the POST method can be used as an implementation host for an extension property adapter. The uri to use for an HTTP adapter is its URL. For example, the /people/person/age property is currently implemented by the Acre program at http://emql.ovalmeta.user.dev.freebaseapps.com/age.
Python adapters are run by the extended MQL service process and are therefore restricted to controlled implementations.
An adapter's role is to fulfill the extended MQL query for the property (or properties) it supports. It communicates with the extended MQL driver by implementing the adapter contract. The adapter defines the semantics of the extension property; extended MQL itself knows nothing about the property except for its name, defining namespace, its adapter URI and which phases to invoke on it.
An extension property adapter is invoked during the different phases necessary to fulfill an extended MQL query. With all phases, JSON is used to format parameters. The same is true for return values except for the help phase which is expected to set the Content-Type response header and return help text.
Depending on the nature of the operations performed by the adapter, there are up to four phases an adapter needs to implement: pre, fetch, reduce and help. By default, implementations for the pre and fetch phase are expected while providing an implementation for the reduce phase is optional. Providing an implementation for the help phase is strongly recommended and ensures the visibility of the property in the extension property gallery.
The phases an adapter implements are listed by the schema for the extension property via the /type/extension/pre, /type/extension/fetch, /type/extension/reduce and /type/extension/help properties. They can be set when creating the property and may be changed later.
An adapter is called once per phase per occurrence of the property it implements in an extended MQL query. An adapter program may implement support for more than one extension property.
The purpose of this phase is to transform the extended MQL query into a valid MQL query suitable for the mqlread service. Each extension property occurrence is replaced by the return value of the pre call on its adapter.
The adapter is invoked with the following parameters:
A dictionary containing the following elements:
For example, in the query:
[{
"name": null,
"type": "/people/person",
"limit": 3,
"age": null,
"age>: 20
}]
the age adapter would be invoked with the following params parameter:
{
"property": "age",
"query": null,
"constraints": [[">", 20]]
}
It is up to the adapter's implementation to make sense, enforce, implement, ignore or reject constraints.
During the pre phase, the adapter's role is to inspect the parent and params parameters to possibly gather some context information about the query. It is expected to return a JSON dictionary that contains any number of clauses to insert into the MQL query in place of the extension property occurrence being processed. This dictionary may be empty. For example, the age property adapter returns:
{
"/people/person/date_of_birth": null,
"/people/deceased_person/date_of_death": null
}
The results for these extra clauses are going to be passed to the adapter during the fetch phase.
If a property's adapter doesn't require a pre phase invocation, the property's /type/extension/pre property must be set to false.
After all extended MQL properties are replaced by pre result clauses, the resulting query is expected to be valid MQL. The extended MQL driver is ready to invoke mqlread with it.
Once the MQL query is run, the results from the pre phase insertions are sent to the corresponding extension property adapters for the fetch phase.
During this phase the adapter is responsible for producing and returning the results it wants inserted in place of the right handside of the original extended MQL property occurrence in the query.
The adapter is invoked with the following parameters:
During the fetch phase, the adapter's role is to iterate the MQL results passed via args and insert a value into the results for each guid it contains. That value can be a simple string, a number or a more complex JSON construct. For example, the age property adapter returns the following for the three results it might have been invoked with:
{
"#9202a8c04000641f8000000000c015ba": 61,
"#9202a8c04000641f80000000008057ef": 45,
"#9202a8c04000641f80000000000765f3": 84
}
The results returned by each property adapter are then inserted in place of the right handside of the original property occurrence in the query's results. If the adapter returned no result for a given match then the match is removed from the overall query's results unless it was optional.
For example, the age property adapter doesn't return an age value for people lacking the data to compute it from.
If a property's adapter doesn't require a fetch phase invocation, the property's /type/extension/fetch property must be set to false.
The reduce phase makes it possible to implement an adapter that returns a value for a given list of query results.
This phase is run after the sort and limit clauses present in the query are effected and thus operates on the final results of the query.
Properties with a reduce phase may occur inside a return clause causing the computed value to replace the list of matches it was computed from. Otherwise, the computed value is inserted into each match.
The adapter is invoked with the following parameters:
The adapter must return a dictionary containing one value keyed on value. See chart adapter for an example.
For example, the /type/object/median property can be applied to a list of numeric results to return its median value:
[{
"name": null,
"type": "/people/person",
"limit": 3,
"age": null,
"age>: 20,
"median": {"value": "age"}
}]
A shorthand syntax makes it possible to compute several such medians in the same subquery by specifying the value to compute it on as a suffix to the median property:
[{
"name": null,
"type": "/people/person",
"limit": 3,
"age": null,
"age>: 20,
"median:age": null
}]
As with MQL's count property, extension properties implemented with a reduce phase can be used in a return clause.
The result of an extended MQL return clause is the dictionary of all the values thus computed. For example:
[{
"name": null,
"type": "/people/person",
"limit": 3,
"age": null,
"age>": 20,
"return": {
"median:age": null,
"min:age": null
"plot:age": {"label": "name"}
}
}]
produces this:
[{
"median:age": 51,
"min:age": 42,
"plot:age": "http://chart.apis.google.com/chart?cht=p&chd=t:51%2C55%2C42&chs=250x100&chl=Jack%20Abramoff%7CBob%20Ney%7CDavid%20Safavian"
}]
If a property's adapter requires a reduce phase invocation, the property's /type/extension/reduce property must be set to true.
The role of this phase is to return documentation about the property and the parameters understood by the adapter implementing it. When help is requested on a property via the extended MQL service the adapter's help phase is invoked.
For example, the URL below returns documentation for the age property: mqlread?help=/people/person/age
The adapter is invoked with the following parameters:
A dictionary containing the following elements:
Instead of returning a JSON string as is the case with the other phases, during the help phase, the adapter is expected to set the HTTP Content-Type response header and write the help text directly into the response. If the content type is text/x-rst, then the extended MQL service will transform the reStructuredText into HTML via docutils before serving it to the requestor.
If a property's adapter does not implement a help phase, the property's /type/extension/help property must be set to false. Properties whose adapter does not implement a help phase are not displayed in the extension property gallery.
extension property matches follow the same semantics as regular MQL properties. Non optional MQL matches that are missing extended MQL matches are eliminated. For example, without cursoring, the query shown below to find some centenarians would not return any results since the first five /people/person matches returned by MQL are very unlikely to be centenarians:
[{
"type": "/people/person",
"limit": 5,
"name": null,
"age": null,
"age>=": 100
}]
The extended MQL service automates cursoring at the top level of the query by going back to MQL for more /people/person matches and re-running the fetch phase until it has found enough centenarians to satisfy the query or MQL returned no more results. By default, it does so in limit sized chunks but this can be rather inefficient if the matches looked for are only rare occurrences in the MQL results. The following query is much faster since it operates on 500-people chunks at a time:
[{
"type": "/people/person",
"limit": 5,
"name": null,
"age": null,
"age>=": 100,
"cursor": 500
}]
At the subquery level though, cursoring is not enabled by default. For example, finding centenarians born in Amsterdam returns no results unless cursoring is explicitely requested at the subquery level:
{
"type": "/location/citytown",
"!/people/person/place_of_birth": [{
"limit": 5,
"id": null,
"name": null,
"age>=": 100,
"age": null,
"cursor": 500
}],
"limit": 1,
"name": "Amsterdam"
}
Such recursive cursoring or crawling can return subtly different results depending on how much cursoring is requested. For example, in addition to cursoring over cities where people are born, the query below will also crawl each city match by re-running the centenarian subquery separately with a MQL cursor until all /people/person matches born in that city are exhausted or enough centenarians are found:
[{
"type": "/location/citytown",
"!/people/person/place_of_birth": [{
"limit": 5,
"id": null,
"name": null,
"age>=": 100,
"age": null,
"cursor": 500
}],
"limit": 5,
"name": null,
"cursor": 20
}]
Whereas this query will only return cities where centenarians where found in the first five MQL matches:
[{
"type": "/location/citytown",
"!/people/person/place_of_birth": [{
"limit": 5,
"id": null,
"name": null,
"age>=": 100,
"age": null
}],
"limit": 5,
"name": null,
"cursor": 20
}]
When requesting subquery cursoring, be sure to fully qualify the properties that introduce the subqueries as they will be inverted when the subqueries are run alone for each containing match.
The cursor directive supports more advanced options that can be specified with a dictionary. For example, to find cities where at least two centenarians were born, the following query can be used:
[{
"type": "/location/citytown",
"!/people/person/place_of_birth": [{
"limit": 5,
"id": null,
"name": null,
"age>=": 100,
"age": null,
"cursor": {"pagesize": 500, "at-least": 2}
}],
"limit": 5,
"name": null,
"cursor": 20
}]
When cursoring is explicitely requested at the top level of the query, the final top level cursor value is returned in the result envelope. This cursor string can then be reused by passing it as a cursor string or a value under the cursor key in the cursor dictionary. If the cursor pagesize was different from the query limit when the cursor was first created, the same pagesize must be reused. For example, the query below returns the next ten people for between the ages of 35 and 45 using the specified cursor with a pagesize of 100:
[{
"type": "/people/person",
"name": null,
"age": null,
"age>=": 35,
"age<=": 45,
"limit": 10,
"cursor": {
"cursor": "cursor:98ad:[o:200][n:199011129]isa:0-199011028:l<-(vip:17071033-199011029:r+9202a8c04000641f800000000000000c->4663)/571436:17072043/@4317121332fd9907599",
"pagesize": 100
},
"sort": "age"
}]
Please note that if not all MQL matches are extended MQL matches, as is the case in the example above, it is likely that some matches will be skipped when the cursor is reused. The cursor uses a period of pagesize and is implicitely reused by the extended MQL engine until limit results are found, the crawling limit was reached or the matches are exhausted. If more than limit results were found after a given cycle, the extra matches are simply discarded. This undesirable behaviour can be controlled by turning off implicit cursor reuse - crawling - and by using the same cursor pagesize and query limit:
[{
"type": "/people/person",
"name": null,
"age": null,
"age>=": 35,
"age<=": 45,
"limit": 10,
"cursor": {
"pagesize": 10,
"crawl": false,
"cursor": "cursor:2fab:[o:10][n:199035379]isa:0-199035219:l<-(vip:17071033-199035220:r+9202a8c04000641f800000000000000c->4663)/3219620:17071574/@4317121335a53d18286"
},
"sort": "age"
}]
The default crawling limit is 100 iterations per cursor and can be changed by setting a cursor's crawl option to the desired limit.
It can sometimes be useful to pass data collected during one phase onto the next. As mentioned earlier, the pre and fetch phases are expected to return JSON dictionaries. By default, the pre phase's results are inserted into the MQL query that is run before the fetch phase and the fetch phase's results are inserted into that query's results. A special key, called :extras is handled differently: its value, which can be any valid JSON, is stored internally and is passed onto the next phase via the params parameter under the :extras key.
For example, the adapter for search uses :extras to store the relevance score of search matches obtained during pre into results during fetch.
Sorting directives that involve only regular MQL properties are passed through to the mqlread service for the usual processing.
On the other hand, sorting on extension property results is done after running the fetch phase before running the limit clause and the reduce phase.
In other words, sorting extended MQL results does not sort Freebase. For example, by sorting on age one only sorts the current extended MQL results and won't easily retrieve the oldest or youngest /people/person instance in Freebase.
The count MQL clause is only supported if the results being counted are only MQL results. Counting the results of a subquery that contains extended MQL property matches with count is not supported at this time.
To count results in an extended MQL query or subquery use the result-count clause. For example, the query below returns the number of distinct cuisines for all the restaurants found in San Francisco:
[{
"!/dining/restaurant/cuisine": [{
"type": "/dining/restaurant",
"/location/location/in": {
"location": "/en/san_francisco",
"limit": 0
}
}],
"id": null,
"return": "result-count"
}]
To submit an extended MQL query you invoke the api/service/mqlread service adding extended=true to the envelope.
The extended MQL service attempts to report all the error information it gets from the services it invokes. For adapters implemented with Acre it helps to pass debug=1 when invoking the extended MQL service.
To see intermediate results between phases, the debug parameter can also be used with one of pre or fetch to stop after that phase was run and return the intermediate results.