A code dive into Mycroft - Jan Newmarch

Overview

Mycroft is an open source digital assistant. Open source means what it says: the code is available and is hackable. So if you want to play with any aspect of Mycroft, you can. The only problem of delving into the source code is of figuring out what is doing what - a standard problem for any piece of complex software.

The Youtube video by Steve Penrod explains the overall structure of Mycroft. The tutorial by JarbasAI goes into more detail about building skills (as well as the standard Mycroft documentation)

But neither of these go down to the code level: what method in which object does what. This document is about that: how do skills get loaded and executed, which objects and which methods get called and what they do. I only cover how skills are loaded and how intents are called and executed.

The two main files are mycroft/skills/core.py and mycroft/skills/intent_service.py, containing the principal classes Mycroft and IntentService. Others will be mentioned as needed.

Debugging

If you want to see what Mycroft is doing, chuck in LOG statements. For example,

LOG.debug('My debug statement')

If Mycroft is run with debugging on (e.g. start-mycroft.sh debug) then the debug statements will appear in /var/log/mycroft/* by default.

A way of figuring out how a particular piece of code was called, is to include a traceback statement:

traceback.print_stack(file = open('/tmp/traceback.txt', 'w'))

You should be also able to use the trace module, but I haven't tried that.

Of course, you have the source code - that's pretty nice to read. But when messages start hopping around on the bus, it can screw your head a bit.

Bus

Mycroft uses a 'bus' to send messages between different components of the system. The 'bus' is actually a web socket: sending a message on the 'bus' is acting as a client to the web socket, while handling received messages is done by the web socket server. These are in files mycroft/messagebus/client/ws.py and mycroft/messagebus/server/ws.py.

You won't need to look at these, save to know that emit() puts a message on the bus and on() registers a handler for a message. What's put on the bus is (serialised) structures of type Message from mycroft/messagebus/message.py.

A Message has an attribute type which is a string, used as the event name to control who handles the message. An example of a serialized message is

{
 "data": {"status": "succeeded"},
 "context": null,
 "type": "skill.handler.complete"
}

Loading intents

In core.py the function load_skill() is responsible for creating a skill. After some sanity checks on a file, it checks if it contains the function create_skill(). If it does, it creates the skill and calls register_decorated() on the skill.

register_decorated() checks each method to see if it is an intent and if so calls register_intent(). This builds the parse data for the intent and puts on the bus a message of type register_intent for the adapt engine, and also calls add_event() with the intent's name and the handler when that intent is executed.

The add_event method adds the generic function wrapper which handles all intent executions. This takes the intent parser's name, and the handler for that intent.

In the meantime, IntentService.handle_register_intent() in intent_service.py handles the register_intent message, and passes the intent's parser data to the adapt parsing engine. It was registered as msg handler for register_intent in IntentService.__init()__

Executing intents

The input recognition loop hands an utterance to IntentService.handle_utterance(). The message data has an attribute utterances which is a list of parsed phrases, typically only one phrase such as ["turn light off"].

This calls _adapt_intent_match() to find the best_intent.

An intent returned from _adapt_intent_match() for the utterance "turn light off" could look like

{   'confidence': 1.0,
    'intent_type': 'lifx-mycroft.samclane:handle_toggle_intent',
    'lifx_mycroft_samclaneOff': 'off',
    'lifx_mycroft_samclaneTarget': 'light',
    'lifx_mycroft_samclaneTurn': 'turn',
    'target': None
    '__tags__': [   {   'confidence': 1.0,
                        'end_token': 0,
                        'entities': [   {   'confidence': 1.0,
                                            'data': [   (   'turn',
                                                            'YeelightTurn'),
                                                        (   'turn',
                                                            'lifx_mycroft_samclaneTurn')],
                                            'key': 'turn',
                                            'match': 'turn'}],
                        'from_context': False,
                        'key': 'turn',
                        'match': 'turn',
                        'start_token': 0},
                    {   'confidence': 1.0,
                        'end_token': 1,
                        'entities': [   {   'confidence': 1.0,
                                            'data': [   (   'light',
                                                            'lifx_mycroft_samclaneTarget')],
                                            'key': 'light',
                                            'match': 'light'}],
                        'from_context': False,
                        'key': 'light',
                        'match': 'light',
                        'start_token': 1},
                    {   'lifx_mycroft_samclaneOff': [   {   'confidence': 1.0,
                                                            'end_token': 2,
                                                            'entities': [   {   'confidence': 1.0,
                                                                                'data': [   (   'off',
                                                                                                'lifx_mycroft_samclaneOff'),
                                                                                                'YeelightOff'),

                                                                                'key': 'off',
                                                                                'match': 'off'}],
                                                            'from_context': False,
                                                            'key': 'off',
                                                            'match': 'off',
                                                            'start_token': 2}]}],


}

This intent has a confidence of 1.0 in its match to the LIFX skill lifx-mycroft.samclane with intent handle_toggle_intent. The first entities section shows that the word turn has matched both the Yeelight and lifx_mycroft_samclane skill's keywords for turn, while the second that the word light has only matched the lifx_mycroft_samclane skill's Target.

This is put back on the bus as part of a reply to the message to handle_utterance, with the message type being the intent's type, that is "type": "lifx-mycroft.samclane:handle_toggle_intent"

This is caught by the wrapper registered to handle intents, and calls the handler for that intent.

In the meantime, if _adapt_intent_match() can't find a best intent, then a message of type intent_failure will be put on the bus. This will be caught by the fallback system.

Messages of type intent_failure are handled by FallbackSkill.make_intent_failure_handler.handler() of core.py. The handler was registered in mycroft/skills/__main__.py by FallbackSkill.make_intent_failure_handler(). This handler loops through all registered fallback handlers until one succeeds. The fallback handlers return True or False to signal success or failure. This is unlike ordinary intent handlers which have no return value (i.e. None).