The purpose of this article is thus to explain how we can “disambiguate” (that is, suggest the proper concept from an ambiguous list) the concepts that have been tagged. What we will do is to show how we can leverage the KBpedia knowledge graph structure as-is to perform this disambiguation. What we will do is to create graph embeddings for each of the KBpedia concepts using the DeepWalk algorithm. Then we will perform simple linear algebra equations on the graph embeddings to determine if the tagged concept(s) is the right one given its context or not. We will test multiple different algorithms and strategies to analyze the impact on the overall disambiguation performance of the system.<\/p>\n
<\/p>\n
[extoc]<\/p>\n
Different Situations<\/h2>\n\nBefore starting to work on the problem, let’s discuss the issues we may encounter with any such concept tagging and disambiguation task. The three possibilities we may encounter are:<\/p>\n
\n- A word in the text refers to a concept, but the concept hasn’t been identified (is not in the graph)<\/li>\n
- A word in the text has been tagged with a single concept, but that concept is not the right one (the right one is not in the graph)<\/li>\n
- A word in the text has been tagged with multiple concepts, so the right concept needs to be selected amongst the options offered.<\/li>\n<\/ol>\n
What we present in this article is how we can automatically disambiguate the situations 2 and 3. (To fix the first situation, not addressed further herein, we would have to add additional surface forms to the knowledge graph by adding new alternative labels to existing concepts, or by adding new concepts with their own surface forms.)<\/p>\n<\/div>\n<\/div>\n
\nDisambiguation Process<\/h2>\n\nThe disambiguation process is composed of two main tasks: the creation of the graph embeddings for each concept in the KBpedia knowledge graph, and then the evaluation of each tagged concept to disambiguate them.<\/p>\n<\/div>\n
\nIntroducing DeepWalk<\/h3>\n\nDeepWalk<\/a> was created to learn social representations<\/i> of a graph’s vertices that capture neighborhood similarity and community membership. DeepWalk generalizes neural language models to process a special language composed of a set of randomly-generated walks.<\/p>\nWith Cognonto, we want to use DeepWalk not to learn social representations<\/i> but to learn the relationship (that is, the similarity) between all of the concepts existing in a knowledge graph given different kinds of relationships such as sub-class-of<\/code>, super-class-of<\/code>, equivalent-class<\/code> or other relationships such as KBpedia’s 80 aspects relationships<\/code>. (Note, we also discussed and used DeepWalk in another Cognonto use case extending KBpedia<\/a>.)<\/p>\nFor this use case we use the DeepWalk algorithm to select concepts from the broader Wikipedia corpus. Other tasks that could be performed using DeepWalk in a similar manner are:<\/p>\n
\n- Content recommendation,<\/li>\n
- Anomaly detection [in the knowledge graph], or<\/li>\n
- Missing link prediction [in the knowledge graph].<\/li>\n<\/ol>\n
Note that we randomly walk the graphs as stated in DeepWalk’s original paper 1<\/a><\/sup>. However more experiments could be performed to change the random walk by other graph walk strategies like depth-first or breadth-first walks.<\/p>\n<\/div>\n<\/div>\n\nCreate Graph Embedding Vectors<\/h3>\n\nWe first create a graph embedding<\/code> for each of the Wikipedia categories. What we have to do is to use the Wikipedia category structure along with the linked KBpedia knowledge graph. Then we have to generate the graph embedding for each of the Wikipedia categories that exist in the structure.<\/p>\nThe graph embeddings are generated using the DeepWalk<\/a> algorithm over that linked structure. It randomly walks the linked graph hundred of times to generate the graph embeddings for each category.<\/p>\n<\/div>\n\nCreate Deeplearning4j Graph<\/h4>\n\nTo generate the graph embeddings, we use Deeplearning4j’s<\/a> DeepWalk implementation. This step creates a Deeplearning4j graph<\/code> structure that is used by its DeepWalk implementation to generate the embeddings.<\/p>\nThe graph we have to create is composed of the latest version of KBpedia knowledge graph version 1.20<\/code>. Since the only concepts we want to disambiguate are the KBpedia reference concepts as tagged by the Cognonto Concept Tagger, then this is the only knowledge structure we have to load to create the graph embeddings.<\/p>\nThen we generate the initial Deeplearning4j graph structure composed of the inferred<\/i> KBpedia knowledge graph.<\/p>\n\n(<\/span>use '[<\/span>cognonto-deepwalk.core<\/span>]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.string<\/span> :as<\/span> string]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.data.csv<\/span> :as<\/span> csv]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.java.io<\/span> :as<\/span> io]<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.query<\/span> :as<\/span> query]<\/span>)<\/span>\n\n(<\/span>defonce<\/span> knowledge-graph<\/span> (<\/span>load-knowledge-graph \"file:\/d:\/cognonto-git\/kbpedia-generator\/kbpedia\/1.10\/target\/kbpedia_reference_concepts_linkage_inferrence_extended_2.n3.owl\"<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nTrain DeepWalk<\/h4>\n\nOnce the Deeplearning4j graph is created, the next step is to create and train the DeepWalk algorithm. What the (create-deep-walk)<\/code> function does is to create and initialize a DeepWalk<\/code> object with the Graph<\/code> we created above and with some hyperparameters.<\/p>\nThe :window-size<\/code> hyperparameter is the size of the window used by the continuous Skip-gram<\/a> algorithm used in DeepWalk. The :vector-size<\/code> hyperparameter is the size of the embedding vectors we want the DeepWalk to generate (it is the number of dimensions of our model). The :learning-rate<\/code> is the initial leaning rate of the Stochastic gradient descent<\/a>.<\/p>\nFor this task, we initially use a window of 15<\/code> and 3<\/code> dimensions to make visualizations simpler to interpret, and an initial learning rate of 2.5%<\/code>.<\/p>\n\n(<\/span>def<\/span> graph<\/span> (<\/span>create-deepwalk-graph knowledge-graph :directed?<\/span> true<\/span>)<\/span>)<\/span>\n\n(<\/span>def<\/span> deep-walk<\/span> (<\/span>create-deep-walk graph \n :window-size<\/span> 15\n :vector-size<\/span> 3\n :learning-rate<\/span> 0.025)<\/span>)<\/span>\n<\/pre>\n<\/div>\nOnce the DeepWalk object is created and initialized with the graph, the next step is to train that model to generate the embedding vectors for each vertex in the graph.<\/p>\n
The training is performed using a random walk iterator. The two hyperparameters related to the training process are the walk-length<\/code> and the walks-per-vertex<\/code>. The walk-length<\/code> is the number of vertices we want to visit for each iteration. The walks-per-vertex<\/code> is the number of timex we want to create random walks for each vertex in the graph.<\/p>\n\n(<\/span>defn<\/span> train<\/span>\n (<\/span>[<\/span>deep-walk iterator]<\/span>\n (<\/span>train deep-walk iterator 1)<\/span>)<\/span>\n (<\/span>[<\/span>deep-walk iterator walks-per-vertex]<\/span>\n (<\/span>.fit<\/span> deep-walk iterator)<\/span>\n (<\/span>dotimes<\/span> [<\/span>n walks-per-vertex]<\/span>\n (<\/span>.reset<\/span> iterator)<\/span>\n (<\/span>.fit<\/span> deep-walk iterator)<\/span>)<\/span>)<\/span>\n (<\/span>[<\/span>deep-walk graph walk-length walks-per-vertex]<\/span>\n (<\/span>train deep-walk (<\/span>new<\/span> RandomWalkIterator<\/span> graph walk-length)<\/span> walks-per-vertex)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\nFor the initial setup, we want to have a walk-length<\/code> of 15<\/code> and we want to iterate the process 175<\/code> times per vertex.<\/p>\n\n(<\/span>train deep-walk graph 15 175)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\nConcept Disambiguation<\/h3>\n\nNow that all of the KBpedia reference concepts are characterized using the graph embeddings, how can we disambiguate each of these concepts in a text? The first thing is to observe what is happening. When we tag a sentence such as “A sandstorm blows over damaged buildings in rebel held area of Douma.”<\/i> the first step that happens is that the surface forms (preferred labels and alternative labels) of the KBpedia concepts are identified within the sentence. As we discussed above, it is possible that a word, or a group of words, may get tagged with multiple KBpedia reference concepts.<\/p>\n
Because we characterized each of the concepts with its graph embeddings, it means that what we are dealing with is a series of vectors that represent the “meaning” of each concept in the knowledge graph structure. What we end-up with is the following:<\/p>\n
\n![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/sentence.png\")
<\/a><\/p>\n<\/div>\nThe next step is to use that information to try to disambiguate each of these concepts. To try to disambiguate the concepts, we have to make a few assumptions. First we have to assume that the graph embeddings created by the DeepWalk algorithm when crawling the KBpedia knowledge graph represent the “meaning”<\/i> of the concept within the knowledge structure. Then we have to assume that the sentence where the concept has been identified creates a context<\/i> that we can use to disambiguate the identified concepts based on their “meaning”<\/i>.<\/p>\n
In the example above, the context is the sentence, and four concepts have been identified. The big assumption we are making here is that each identified concept is “tightly” related to others in the knowledge graph. Given this assumption, what we have to do is to calculate the relatedness of each concept, within the context, and keep the ones that are closest to each other.<\/p>\n<\/div>\n
\nGold Standard Creation<\/h4>\n\nBefore digging into how we will perform the disambiguation of these concepts, another step is to create a gold standard that we will use to evaluate the performance of our model and to check the impact of different hyperparameters on the disambiguation process. We created this gold standard manually by using random sentences found in online news articles. We first tagged each of these sentences using the Cognonto Concept Tagger. Then, we manually disambiguated each of the tagged concepts. This gives us the properly labeled training set for our exercise. The result is the following gold standard file<\/a>.<\/p>\nEach of the annotations looks like this:<\/p>\n
![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_7c24ba70dbba722f3a2b8cfcf3afea2446f60105-768x18.png\")
<\/a><\/p>\nThis markup first presents the surface forms for the related concept(s) in the knowledge graph within the double brackets, followed by the concept(s) URI endings between the double parenthesis. The double colon ::<\/code> designator provides the suggested disambiguated concept. If nothing follows the double colon it means that the correct concept does not exist in the knowledge graph).<\/p>\n<\/div>\n<\/div>\n\nDisambiguation Method<\/h4>\n\nThe actual disambiguation method is based on some simple linear algebra formulas.<\/p>\n
Each tagged word\u00c2\u00a0![](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_0d1399fad7c1911e6241538c7b4fdfb50329d273.png\")
<\/a> is related to a sliding context\u00c2\u00a0![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_7d61ea454e712ce09c7f75d3140788acf53ae5fe.png\")
<\/a> where ![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_9783bbde9fb6923e6ceddef20e7a918b54a2f992.png\")
<\/a>. Each tagged word has\u00c2\u00a0![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_ef038ba123ebef4319d569d54ec40dc5f67bc0c5.png\")
<\/a> graph embedding vectors (i.e. a “sense”<\/i> vector)\u00c2\u00a0![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_a5f0e1561072b96080a9c732794bd18fe00f1a57.png\")
<\/a> where ![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_72cf9483351da664caaca6cbde55432080d6c202.png\")
<\/a>. What we want the disambiguation method to do is to calculate a score between each embedding vector of each identified concept of each word given its sliding context. The score is calculated by performing the dot product between the graph embedding vector\u00c2\u00a0![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_6b526e3a714ac2c7be46661934ef653eac7e7fd8.png\")
<\/a> and the sliding context vector ![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_774d0389f3a45245568f0233190a445a7ad65f53.png\")
<\/a>. The sliding context vector is calculated by summing the graph embedding vector\u00c2\u00a0<\/a> of each word\u00c2\u00a0![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_ccb5e9230532a6990c88b620742d233da163504d.png\")
<\/a> within the window:<\/p>\n![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_73e561775d087190d01c276973ae7b6941ab72e7.png\")
<\/a><\/p>\nFinally the score is calculated using:<\/p>\n
![\"\"](\"https:\/\/fgiasson.com\/blog\/wp-content\/uploads\/2017\/01\/kbpedia-concepts-disambiguation_mkb_15bde81280c85f78a2f16bd40d9ac16d62b8fb41.png\")
<\/a><\/p>\nThis is the first definition of the score we want to use to begin to disambiguate concepts from the knowledge graph associated with words in a sentence. As you can read from the formula, only the first sense for each word\u00c2\u00a0<\/a> is being selected for each word within the sliding window that creates the context. (Other variants are be explored later in this article.) However as we will see below, we will change that formulas a bit such that the score becomes easier to understand for a human by expressing it in degrees. What is important to understand here is that we use two vectors to calculate the score: the graph embedding of the concept we want to disambiguate and the vector calculated from the windowed context that sums all the graph embedding vectors of each concept within that window.<\/p>\nNow let’s put this method into code. The first thing we have to do is to create a lookup table that is composed of the graph embeddings for each of the concepts that exist in the KBpedia knowledge graph as we calculated with the DeepWalk algorithm above.<\/p>\n
\n(<\/span>def<\/span> index<\/span> (<\/span>->><\/span> (<\/span>query<\/span>\/<\/span>get-classes knowledge-graph)<\/span>\n (<\/span>map-indexed (<\/span>fn<\/span> [<\/span>i class]<\/span>\n (<\/span>if-not<\/span> (<\/span>string? class)<\/span>\n {<\/span>(<\/span>.toString<\/span> (<\/span>.getIRI<\/span> class)<\/span>)<\/span> (<\/span>inc i)<\/span>}<\/span>\n {<\/span>class (<\/span>inc i)<\/span>}<\/span>)<\/span>)<\/span>)<\/span>\n (<\/span>apply merge)<\/span>)<\/span>)<\/span>\n\n(<\/span>def<\/span> ^:dynamic<\/span> vertex-vectors<\/span> (<\/span>.getVertexVectors<\/span> (<\/span>.lookupTable<\/span> deep-walk)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\nGetting the vector\u00c2\u00a0<\/a> for the word\u00c2\u00a0<\/a> is as simple as getting it from the lookup table we created above:<\/p>\n\n
Before starting to work on the problem, let’s discuss the issues we may encounter with any such concept tagging and disambiguation task. The three possibilities we may encounter are:<\/p>\n
- \n
- A word in the text refers to a concept, but the concept hasn’t been identified (is not in the graph)<\/li>\n
- A word in the text has been tagged with a single concept, but that concept is not the right one (the right one is not in the graph)<\/li>\n
- A word in the text has been tagged with multiple concepts, so the right concept needs to be selected amongst the options offered.<\/li>\n<\/ol>\n
What we present in this article is how we can automatically disambiguate the situations 2 and 3. (To fix the first situation, not addressed further herein, we would have to add additional surface forms to the knowledge graph by adding new alternative labels to existing concepts, or by adding new concepts with their own surface forms.)<\/p>\n<\/div>\n<\/div>\n
\nDisambiguation Process<\/h2>\n
\nThe disambiguation process is composed of two main tasks: the creation of the graph embeddings for each concept in the KBpedia knowledge graph, and then the evaluation of each tagged concept to disambiguate them.<\/p>\n<\/div>\n
\nIntroducing DeepWalk<\/h3>\n
\nDeepWalk<\/a> was created to learn social representations<\/i> of a graph’s vertices that capture neighborhood similarity and community membership. DeepWalk generalizes neural language models to process a special language composed of a set of randomly-generated walks.<\/p>\n
With Cognonto, we want to use DeepWalk not to learn social representations<\/i> but to learn the relationship (that is, the similarity) between all of the concepts existing in a knowledge graph given different kinds of relationships such as
sub-class-of<\/code>,super-class-of<\/code>,equivalent-class<\/code> or other relationships such as KBpedia’s80 aspects relationships<\/code>. (Note, we also discussed and used DeepWalk in another Cognonto use case extending KBpedia<\/a>.)<\/p>\nFor this use case we use the DeepWalk algorithm to select concepts from the broader Wikipedia corpus. Other tasks that could be performed using DeepWalk in a similar manner are:<\/p>\n
- \n
- Content recommendation,<\/li>\n
- Anomaly detection [in the knowledge graph], or<\/li>\n
- Missing link prediction [in the knowledge graph].<\/li>\n<\/ol>\n
Note that we randomly walk the graphs as stated in DeepWalk’s original paper 1<\/a><\/sup>. However more experiments could be performed to change the random walk by other graph walk strategies like depth-first or breadth-first walks.<\/p>\n<\/div>\n<\/div>\n
\nCreate Graph Embedding Vectors<\/h3>\n
\nWe first create a
graph embedding<\/code> for each of the Wikipedia categories. What we have to do is to use the Wikipedia category structure along with the linked KBpedia knowledge graph. Then we have to generate the graph embedding for each of the Wikipedia categories that exist in the structure.<\/p>\nThe graph embeddings are generated using the DeepWalk<\/a> algorithm over that linked structure. It randomly walks the linked graph hundred of times to generate the graph embeddings for each category.<\/p>\n<\/div>\n
\nCreate Deeplearning4j Graph<\/h4>\n
\nTo generate the graph embeddings, we use Deeplearning4j’s<\/a> DeepWalk implementation. This step creates a Deeplearning4j
graph<\/code> structure that is used by its DeepWalk implementation to generate the embeddings.<\/p>\nThe graph we have to create is composed of the latest version of KBpedia knowledge graph version
1.20<\/code>. Since the only concepts we want to disambiguate are the KBpedia reference concepts as tagged by the Cognonto Concept Tagger, then this is the only knowledge structure we have to load to create the graph embeddings.<\/p>\nThen we generate the initial Deeplearning4j graph structure composed of the inferred<\/i> KBpedia knowledge graph.<\/p>\n
\n(<\/span>use '[<\/span>cognonto-deepwalk.core<\/span>]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.string<\/span> :as<\/span> string]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.data.csv<\/span> :as<\/span> csv]<\/span>)<\/span>\n(<\/span>require '[<\/span>clojure.java.io<\/span> :as<\/span> io]<\/span>)<\/span>\n(<\/span>require '[<\/span>cognonto-owl.query<\/span> :as<\/span> query]<\/span>)<\/span>\n\n(<\/span>defonce<\/span> knowledge-graph<\/span> (<\/span>load-knowledge-graph \"file:\/d:\/cognonto-git\/kbpedia-generator\/kbpedia\/1.10\/target\/kbpedia_reference_concepts_linkage_inferrence_extended_2.n3.owl\"<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nTrain DeepWalk<\/h4>\n
\nOnce the Deeplearning4j graph is created, the next step is to create and train the DeepWalk algorithm. What the
(create-deep-walk)<\/code> function does is to create and initialize aDeepWalk<\/code> object with theGraph<\/code> we created above and with some hyperparameters.<\/p>\nThe
:window-size<\/code> hyperparameter is the size of the window used by the continuous Skip-gram<\/a> algorithm used in DeepWalk. The:vector-size<\/code> hyperparameter is the size of the embedding vectors we want the DeepWalk to generate (it is the number of dimensions of our model). The:learning-rate<\/code> is the initial leaning rate of the Stochastic gradient descent<\/a>.<\/p>\nFor this task, we initially use a window of
15<\/code> and3<\/code> dimensions to make visualizations simpler to interpret, and an initial learning rate of2.5%<\/code>.<\/p>\n\n(<\/span>def<\/span> graph<\/span> (<\/span>create-deepwalk-graph knowledge-graph :directed?<\/span> true<\/span>)<\/span>)<\/span>\n\n(<\/span>def<\/span> deep-walk<\/span> (<\/span>create-deep-walk graph \n :window-size<\/span> 15\n :vector-size<\/span> 3\n :learning-rate<\/span> 0.025)<\/span>)<\/span>\n<\/pre>\n<\/div>\nOnce the DeepWalk object is created and initialized with the graph, the next step is to train that model to generate the embedding vectors for each vertex in the graph.<\/p>\n
The training is performed using a random walk iterator. The two hyperparameters related to the training process are the
walk-length<\/code> and thewalks-per-vertex<\/code>. Thewalk-length<\/code> is the number of vertices we want to visit for each iteration. Thewalks-per-vertex<\/code> is the number of timex we want to create random walks for each vertex in the graph.<\/p>\n\n(<\/span>defn<\/span> train<\/span>\n (<\/span>[<\/span>deep-walk iterator]<\/span>\n (<\/span>train deep-walk iterator 1)<\/span>)<\/span>\n (<\/span>[<\/span>deep-walk iterator walks-per-vertex]<\/span>\n (<\/span>.fit<\/span> deep-walk iterator)<\/span>\n (<\/span>dotimes<\/span> [<\/span>n walks-per-vertex]<\/span>\n (<\/span>.reset<\/span> iterator)<\/span>\n (<\/span>.fit<\/span> deep-walk iterator)<\/span>)<\/span>)<\/span>\n (<\/span>[<\/span>deep-walk graph walk-length walks-per-vertex]<\/span>\n (<\/span>train deep-walk (<\/span>new<\/span> RandomWalkIterator<\/span> graph walk-length)<\/span> walks-per-vertex)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\nFor the initial setup, we want to have a
walk-length<\/code> of15<\/code> and we want to iterate the process175<\/code> times per vertex.<\/p>\n\n(<\/span>train deep-walk graph 15 175)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\nConcept Disambiguation<\/h3>\n
\nNow that all of the KBpedia reference concepts are characterized using the graph embeddings, how can we disambiguate each of these concepts in a text? The first thing is to observe what is happening. When we tag a sentence such as “A sandstorm blows over damaged buildings in rebel held area of Douma.”<\/i> the first step that happens is that the surface forms (preferred labels and alternative labels) of the KBpedia concepts are identified within the sentence. As we discussed above, it is possible that a word, or a group of words, may get tagged with multiple KBpedia reference concepts.<\/p>\n
Because we characterized each of the concepts with its graph embeddings, it means that what we are dealing with is a series of vectors that represent the “meaning” of each concept in the knowledge graph structure. What we end-up with is the following:<\/p>\n
\n<\/a><\/p>\n<\/div>\nThe next step is to use that information to try to disambiguate each of these concepts. To try to disambiguate the concepts, we have to make a few assumptions. First we have to assume that the graph embeddings created by the DeepWalk algorithm when crawling the KBpedia knowledge graph represent the “meaning”<\/i> of the concept within the knowledge structure. Then we have to assume that the sentence where the concept has been identified creates a context<\/i> that we can use to disambiguate the identified concepts based on their “meaning”<\/i>.<\/p>\n
In the example above, the context is the sentence, and four concepts have been identified. The big assumption we are making here is that each identified concept is “tightly” related to others in the knowledge graph. Given this assumption, what we have to do is to calculate the relatedness of each concept, within the context, and keep the ones that are closest to each other.<\/p>\n<\/div>\n
\nGold Standard Creation<\/h4>\n
\nBefore digging into how we will perform the disambiguation of these concepts, another step is to create a gold standard that we will use to evaluate the performance of our model and to check the impact of different hyperparameters on the disambiguation process. We created this gold standard manually by using random sentences found in online news articles. We first tagged each of these sentences using the Cognonto Concept Tagger. Then, we manually disambiguated each of the tagged concepts. This gives us the properly labeled training set for our exercise. The result is the following gold standard file<\/a>.<\/p>\n
Each of the annotations looks like this:<\/p>\n
<\/a><\/p>\nThis markup first presents the surface forms for the related concept(s) in the knowledge graph within the double brackets, followed by the concept(s) URI endings between the double parenthesis. The double colon
::<\/code> designator provides the suggested disambiguated concept. If nothing follows the double colon it means that the correct concept does not exist in the knowledge graph).<\/p>\n<\/div>\n<\/div>\n\nDisambiguation Method<\/h4>\n
\nThe actual disambiguation method is based on some simple linear algebra formulas.<\/p>\n
Each tagged word\u00c2\u00a0
<\/a> is related to a sliding context\u00c2\u00a0<\/a> where <\/a>. Each tagged word has\u00c2\u00a0<\/a> graph embedding vectors (i.e. a “sense”<\/i> vector)\u00c2\u00a0<\/a> where <\/a>. What we want the disambiguation method to do is to calculate a score between each embedding vector of each identified concept of each word given its sliding context. The score is calculated by performing the dot product between the graph embedding vector\u00c2\u00a0<\/a> and the sliding context vector <\/a>. The sliding context vector is calculated by summing the graph embedding vector\u00c2\u00a0<\/a> of each word\u00c2\u00a0<\/a> within the window:<\/p>\n<\/a><\/p>\nFinally the score is calculated using:<\/p>\n
<\/a><\/p>\nThis is the first definition of the score we want to use to begin to disambiguate concepts from the knowledge graph associated with words in a sentence. As you can read from the formula, only the first sense for each word\u00c2\u00a0
<\/a> is being selected for each word within the sliding window that creates the context. (Other variants are be explored later in this article.) However as we will see below, we will change that formulas a bit such that the score becomes easier to understand for a human by expressing it in degrees. What is important to understand here is that we use two vectors to calculate the score: the graph embedding of the concept we want to disambiguate and the vector calculated from the windowed context that sums all the graph embedding vectors of each concept within that window.<\/p>\nNow let’s put this method into code. The first thing we have to do is to create a lookup table that is composed of the graph embeddings for each of the concepts that exist in the KBpedia knowledge graph as we calculated with the DeepWalk algorithm above.<\/p>\n
\n(<\/span>def<\/span> index<\/span> (<\/span>->><\/span> (<\/span>query<\/span>\/<\/span>get-classes knowledge-graph)<\/span>\n (<\/span>map-indexed (<\/span>fn<\/span> [<\/span>i class]<\/span>\n (<\/span>if-not<\/span> (<\/span>string? class)<\/span>\n {<\/span>(<\/span>.toString<\/span> (<\/span>.getIRI<\/span> class)<\/span>)<\/span> (<\/span>inc i)<\/span>}<\/span>\n {<\/span>class (<\/span>inc i)<\/span>}<\/span>)<\/span>)<\/span>)<\/span>\n (<\/span>apply merge)<\/span>)<\/span>)<\/span>\n\n(<\/span>def<\/span> ^:dynamic<\/span> vertex-vectors<\/span> (<\/span>.getVertexVectors<\/span> (<\/span>.lookupTable<\/span> deep-walk)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\nGetting the vector\u00c2\u00a0
<\/a> for the word\u00c2\u00a0<\/a> is as simple as getting it from the lookup table we created above:<\/p>\n\n