Part 3: Migrating Chatbot from CSV to JSON

Now that we have shown how questions and responses can be stored and fetched dynamically from a CSV file, we should move to something more structured for containing the data.

JSON stands for JavaScript Object Notation and a basic definition and description of it can be found here. The JSON structure allows us to write the questions and responses in the same way that we did for the CSV file, but we can easily add more structure and data types into the structure, converting it from a flat CSV row into something that we can treat more like a data object.

First, a great tool that helped point out a few mistakes in my JSON document: JSONLint.

The JSON document:

{
    "chatbotdata": [{
            "id": 1,
            "question": "Hello. May I have your full name? (last_name first_name)",
            "prior_response": "",
            "responses": [{
                "id": 1,
                "response": "Thank you"
            }]
        },
        {
            "id": 2,
            "question": "Would you like to 1) start a new claim 2) check on the status of an existing claim 3) speak with a customer service representative or 4) something else?",
            "prior_response": "Thank you",
            "responses": [{
                    "id": 1,
                    "response": "OK. I can help you with that."
                },
                {
                    "id": 2,
                    "response": "OK. I need to get some more information from you."
                },
                {
                    "id": 3,
                    "response": "OK. Let me connect you with an agent."
                },
                {
                    "id": 4,
                    "response": "OK. Please provide more information."
                }
            ]
        }
    ]
}

Next, the test case...with a few adjustments:

def test_second_question_from_response(self):
    lines = cb3.read_json_file() # read the file
    q1 = cb3.get_question_from_response(lines, "") # find line
    r1 = cb3.get_responses_for_question(lines, q1)
    q2 = cb3.get_question_from_response(lines, r1)
    r2 = cb3.get_responses_for_question(lines, q2)

    first_phrase = q2.split(")")
    self.assertEqual(first_phrase[0], "Would you like to 1")

The code turns out a bit cleaner because we can now use the structure to look values up more easily:

import json

def read_json_file():
    filename = "chatbot.json"
    with open(filename, "r") as chatbotfile:
        data = json.load(chatbotfile)
    return data   


def get_question_from_response(lines, response):

    for line in lines['chatbotdata']:
        if response == line['prior_response']:
            return line['question']
        else:
            pass


def get_responses_for_question(lines, question):
    responses = ""
    for line in lines['chatbotdata']:
        if question == line['question']:
            for response in line['responses']:
                responses += response['response']
        else:
            pass

    return responses

Chatbot 2: test run

We could now write a method to interact with the JSON document to take the input from one question to fetch the question corresponding to that particular answer and continue to ping-pong between the user input and the JSON document until the user has worked their way through the entire document. However, instead of using the string response to retrieve the string question, we can search by IDs and retrieve the question corresponding to the question_id of that particular response.

The code would change to the following:


def get_question_from_response(lines, response):
    for line in lines['chatbotdata']:
        try:
            if response == line['id']:
                return line['question']
            else:
                pass
        except ValueError:
            if response == line['prior_response']:
                return line['question']
            else:
                pass

and the unit test:


def test_questions_from_ids(self):
    lines = cb3.read_json_file() # read the file
    q1 = cb3.get_question_from_response(lines, 1) # find line
    r1 = cb3.get_responses_for_question(lines, 1)
    q2 = cb3.get_question_from_response(lines, 2)
    r2 = cb3.get_responses_for_question(lines, 2)
    first_phrase = q2.split(")")
    self.assertEqual(first_phrase[0], "Would you like to 1")

Chatbot 2: test run

I know that I missing some fundamental logic that will allow the user to enter a value, we look it up in the JSON, retrieve the corresponding ID(s) and retrieve the next question. I have intentionally left those out for now because I am focused now on just proving that it can be done. I believe that I have shown enough that it CAN be done and explained how to take this full circle, but for now, I think we have done enough because we are building towards something much bigger and better and I don't want to spend too much time dwelling on something that I am going to ultimately throw away.

We could detour here and show the same steps can be done in a database (like SQLite), but I think the JSON adequately covers how this would look. Instead of a JSON document, it would be a couple of tables, one for questions and one for responses. The responses would be a many-to-one relationship with the question table, and we would make the same calls to the database as we did with the JSON document.

I think here is a good point to move onto some of the AI logic libraries and see if we can move our logic into those packages.