PSU ECE Robotics 2 Class Project

PSU ECE Robotics 2 Class Project

Tony Muilenburg, Dawson Green

Computer vision and control

Baby Schrodinger Cat (CatBot)

Contents

Project summary

Robot cat

Robot firmware code

Robot control code

Computer vision code

Next steps

CatBot Control Introduction

Main Execution File (main.cpp)

Compilation Warnings

Namespaces

Global Definitions File (definitions.h)

#define values

Use of enums

Use of ref structs & ref classes

Port Manager Object (PortManager.h & PortManager.cpp)

Port Available and Robot Ready functions

Sending BT Messages to an NXT

SerialPort object

Catbot Auto Control object (CatbotAutoCtrl.h & CatbotAutoCtrl.cpp)

FilePath variable

Reading OpenCV data

Calculating Catbot Motion

Application UI (AppUI.h, AppUI.cpp & AppUI_Events.cpp)

Functions for the Setup Tab

Manual Controls

Automatic Control

Project summary

With the increase in speed of personal computers many things have become possible. One of the most interesting applications is computer vision. For this project we used computer vision to control a robot as part of a robot theater program Dr. Marek Perkowski putting together(Link). Using a single overhead camera we were able to control a robot, tracking its position and angle, and have it autonomously drive toward an arbitrary target location. This report will describe the steps necessary to reproduce what we did, as well as cover the materials used.

Robot cat

Because another team was working on the Schrodinger cat robot, we decided to create use a Lego Mindstorms robot, and added a baby kitten on top of it. The base for the robot came from the 9797 First Lego League kit. This kit is described on Lego’s website at this link:9797. Instructions for assembling the robot can be found here: robot base instructions. To simplify tracking of the robot, a green circle, and a pink square were taped to the top to the baby kitten toy. Having two different colors allowed us to calculate the direction the cat was facing. The cat riding the robot is shown below before the colors were added.

Robot firmware code

Communication with the robot is achieved using Bluetooth. The Lego NXT robot is equipped with Bluetooth, though sending commands using a laptop is not possible using Lego’s standard code. To communicate with the NXT and set it up to accept commands, the firmware on the NXT brick is first overwritten using a program called RobotC. This program can be used for free for ten days on a computer, after which the program must be purchased for $50. Fortunately, once the firmware has been written to the robot, RobotC no longer needs to be used.

Robot control code

The code used to send commands to the robot was written in visual studio using visual c++, and is called catbot control. The following figure outlines the project structure:

The user interface is split up into two tabs. The first allows the user select the Bluetooth device that is to be used, and configure it. While some laptops come with integrated Bluetooth, the laptop used for this did not, so a USB dongle was added which added it. After selecting the Bluetooth port, the connect button will enable communication with the NXT equipped robot that has been flashed with the robotc firmware created for this project. The computer vision code uses a text file to pass location data to this robot control code. The path to the file the computer vision code generates should be added to the textbox using the find file button which pops up a file browse dialog window. The “Test File” button should then be used to ensure that the path is valid and file permissions are ok (permission to read the file is granted).

The second tab has buttons to allow for manual or automatic control. In manual mode, the robot can be turned, or orders can be given for it to go forward or backward. A speed control is available, as well as an emergency stop (see below).

In automatic mode, the code will read the coordinate from the file the computer vision code generated, then a heading and distance will be generated from the robot to a blue dot, and commands will be send to have the robot turn and drive to the blue dot. The coordinates are written and read many times per second to ensure that the robot does not drive too far, or get off course.

The code for this project can be downloaded from this link: CatBot control code

Computer vision code

By default, the compute vision code (cat tracking opencv) opens many windows. The following figure show what screen looks like in debug mode.

The top four windows show the threshold image for a particular color. The colors red, blue, green and pink are tracked, though red is not currently in use (see the next steps section for more on this). In looking at the green and pink threshold windows, it is clear that these colors exist in the camera feed. All other colors are filtered out by specifying filter in HSV format (follow this link for a description of HSV). Along the bottom, there are four windows each of which have sliders that can be used to adjust the H, S, and V values and filter out all but the desired color to be tracked. The “camera feed” window in the middle of the lower portion of the screen shows what the camera sees along with superimposed circles that correspond to areas being tracked.

The code writes the coordinates of each of the discovered colors to a file called “cat_tracking.txt” which is then read by the cat control code, which guides the cat to the desired location. The format of the file is given below:

Next steps

While many of the next steps are outlined in Dr. Perkowski’s robot theatre script, some of the most obvious next steps include:

• Put together a third option for scripted control in addition to manual and automatic control of the baby Schrodinger cat.
• Enable tracking and control of other robots, including Schrodinger cat, Einstein, Newton, Bohr, Marie Curie, etc.
• Enable finer granularity for the fuzzy logic speed control
• Incorporate depth control using a Kinect camera
• Train and use a Haar cascade classifier to recognize robots, and robot orientation (color would no longer be needed for tracking).

CatBot Control Introduction

This document describes the breakdown of files used in the CatbotControl program for the baby Schrodinger Cat robot (referred to as catbot throughout). Each section of the project is broken down in turn and notes on the usage and setup configuration for it are discussed. Be aware that full understanding of the code will likely only come from reading it and experimenting with it. However, this should hopefully help expedite that process along. If any objects or styles of code are unfamiliar to you when you come across them, please refer to the Microsoft MSDN for details about Visual C++ and the Common Language Runtime (CLR).

NOTE: This document assumes you are using Visual Studio Professional 2013. If you are using a different IDE to compile this project, directions to edit settings may be different. Please use your best judgement to interact with your IDE UI and update this document if future classes will use a different program to interact with this code.

Main Execution File (main.cpp)

The main routine is cleverly hidden in the main.cpp file. This file is fairly barebones and is a useful format for how to invoke a GUI in Visual C++. The file does contain a few notes on project settings, so make sure these are configured correctly before compiling the code. To edit them, right click the project in the solution explorer and select “Properties”. This opens the master set of project properties for modification.

Compilation Warnings

Any time the code produces are warning for C4101, you are free to ignore it. This warning is generated by the compiler letting you know that a local variable isn’t actually used in its function and most often caused by try/catch blocks. If in doubt, check the line of code throwing the warning. There will be a comment if it was expected and should be ignored.

Namespaces

When possible, this code declares and uses namespaces to shorten down conventions. However, for accuracy, this document will often expand out the namespace of an object so it’s clear what object is being discussed. Namespaces are separated by the double colon indicator “::”, with the last keyword on the line being the actual object in question. Please check Google if you need more information on how these work.

Global Definitions File (definitions.h)

The global definition file is included in all other header files. It is used as a centralized hub for all file includes, all #defines and all the global variables in this namespace. Note that #defines aren’t restricted by Visual C++ to a particular namespace, so care should be exercised about naming conventions for them.

#define values

Visual C++ doesn’t provide a way to define string constants when using CLR. While most numerical values have been locked in as globals, we have not found a good way around using #defines to drop in strings to path files yet. Please update this section if you find a way.

Use of enums

If you’re not familiar with enums, they provide a good way to turn keywords into integers. Each enum defaults to numbering its elements from 0 to length-1. While you can manually order these and change the numbering system, this isn’t encouraged. It is expressly prohibited for the BotIDenum, as the TOTAL_BOTS value would then be inaccurate. Please refer to Google if you need more information on how these work.

Use of ref structs & ref classes

The really cool thing about Visual C++ is that it has a garbage collector available to it that’s similar to other garbage collected languages. However, this causes a split between “managed” code and “native” code. Managed objects can always be identified by the hat (“^”) character at the end of their declaration (“Object^”). If all references to an object are deleted by destructor or go out of scope of the code, then the objects destructor will be automatically invoked. You can create your own managed objects simply by putting the keyword “ref” before the declaration of any struct or class you create. For the most part, this project uses them to make life easier and more automagical.

Port Manager Object (PortManager.h & PortManager.cpp)

This object provides a useful abstraction for interacting with the catbot and any future robots you choose to tie into this project. At its core, it is designed to manage all of the serial ports (COM ports) used to communicate with robots controlled by this application. From the outside, a user simply passes the appropriate BotID and this object will handle sending or receiving data to the right robot. While only catbot is currently established in the code right now, stubs are provided to show how the functions can be expanded to control a full array of robots. NOTE: This object can only interact with robots registered to a Windows COM port. If you have another communication system, you’ll need a new class.

Port Available and Robot Ready functions

While the NXT provides the Bluetooth Serial Port Profile (SPP) that allows it to be controlled over a COM port when paired to the computer, the BT receiver on it is fairly slow. It’s so slow, in fact, that testing it for availability is a prohibitively time expensive operation. Therefore, this code will only check if ports are currently owned by this program and configured for this program’s robots before trying to interact with them. If the port gets hijacked by another process, this code won’t realize it and will probably start doing weird things.

Sending BT Messages to an NXT

While the catbot has its own code to interpret the messages to the NXT and translate them into motions, the NXT itself requires messages to be formatted in specific ways. Each communication is referred to as a “telegram” and a master list can be found in the supplementary documentation for RobotC. For this project, each message can be broken down and understood in the following way:

arrayByte>^BT_Msg=gcnewarrayByte{0x07,0x00,0x80,0x09,0x00,0x03,msg,msgMag,0x00};

1. Variable type (“array<Byte>^”) – Managed array of 8 bit bytes gets automatically disposed of after sending.
2. Message Length (“0x07, 0x00”) – Each message must start with the length of the actual payload, LSB first. This format indicates that there are 7 bytes being sent not including the two bytes declaring the length.
3. Command Type (“0x80”) – This command doesn’t need a response from the NXT.
4. Command (“0x09”) – Our code is writing a message to the system
5. Message Mailbox (“0x00”) – The message will be in “mailbox1” for the RobotC code to read from.
6. Message Size (“0x03”) – Number of bytes of actual message data (i.e. length of the remainder of the array)
7. Message Contents (“msg, msgMag, 0x00”) – The RobotC code specifies that the robot command and command speed are bytes 1 and 2 of the message. The NXT requires all messages to terminate with a null character, so “0x00” must be tacked on to the end of any message and included in the payload calculations.

Please refer to “Appendix 2 – LEGO MINDSTORMS NXT Direct Commands.pdf” for details of how to format and send other messages to the NXT.

SerialPort object

Most of the structure of this code is making use of functions and properties provided by the System::IO::Ports::SerialPort object. MSDN has excellent documentation about the properties of this object and you’re encouraged to check it out if you need more info on an object function call.

Catbot Auto Control object (CatbotAutoCtrl.h & CatbotAutoCtrl.cpp)

Just as the PortManager helped abstract the interface layer for communicating with the robot, the Auto Control object is designed to abstract the process of reading data from the OpenCV program and determining what to do with that data. Because each robot will consume data differently, this object is only targeted to the catbot. However, it should provide a good template if you want other robots to consume OpenCV code in a similar fashion.

FilePath variable

While the object used to read data from the file in readData() can theoretically support relative paths to the file, it is our experience that this can cause some odd read/write issues with the file. These were fixed by using the absolute path of the variable. If you can ever figure out why, please update this document with the answers.

Reading OpenCV data

The OpenCV program takes measurements of position of the necessary markers and provides those in a valid text file. The GUI can find this file in the system and test that it is able to be read, but reading that file requires a set of keys. Back in definitions.h, there’s a series of keywords that corresponds to the data relevant to catbot from OpenCV. Note that ORDERING IS IMPORTANT for the moment, as each keyword is mapped to a specific variable in the data reading. The order the keywords appear in the text file doesn’t matter (or if unexpected keywords are there), but the order in the definitions.h file does.

Each keyword expects an integer to be read from the textfile, separated by any of the valid delimiters found in definitions.h. Currently, a space (‘ ‘) is used. This program will use position data to calculate angles, so the more accurate those integers can be, the better. Values are currently scaled to the number of pixels viewed by the webcam.

Calculating Catbot Motion

The algorithm works basically by calculating differences between the target and the current position/heading of the catbot. Angles are constrained to being between -180 and +180 degrees, with conversions being provided to transform radians into degrees. The process is that, if the catbot is “close enough”, it will stop. If it is in the right general direction, it will drive forward. If it still needs to turn, then it will turn in the “fastest” direction. Catbot motion resulting from this calculation are then packaged and returned to the caller. NOTE: the calculation will return a null pointer if the read OpenCV data was invalid or if the calculation causes a repeat in the motion being sent to the catbot. The caller is expected to NOT send data if it gets a null pointer back.