Posts

Spring 2016 Pathfinder: Project Tango Bluetooth API

tango_system

by:

Peiyuan Xu (Project Manager)

with contribution from:

Qianyi Tang (CSULB Computer Science Major Graduate Student)

Table of Contents

Introduction:

In our Project Tango system block diagram, one of the software module is to develop Bluetooth API in the Point Cloud App command sequence. This blog post is to show the approach of modifying point cloud App to pair and connect another device to Tango tablet through Bluetooth while still running Point Cloud.

Steps:

  1. Download and install Android Studio and Tango SDK
  2. Download the Point Cloud sample code and import it into Android Studio
  3. Connect the Tango tablet to the PC and enable the Android Debugging Bridge (ADB)
  4. Build and run the Point Cloud sample code. This would automatically install the App in Tango tablet
  5. Open project → app → Java → PointCloudActivity.java
  6. Open project → res → layout → activity_jpoint_cloud.xml
  7. Open project → res → values → strings.xml
  8. Breakdown and modify the code

The download instruction and links can be found here:

 

Code Breakdown and explanations:

1.Create the Bluetooth Button

Add the following code to “activity_jpoint_cloud.xml”
/******/
<Button
android:id=“@+id/bluetooth_button”
android:layout_width=“100dp”
android:layout_height=“wrap_content”
android:layout_above=“@+id/first_person_button”
android:layout_alignParentRight=“true”
android:layout_marginBottom=“5dp”
android:layout_marginRight=“5dp”
android:paddingRight=“5dp”
android:text=“@string/bluetooth” />

/******/

The code above will generate a button on the screen called”bluetooth” with the width of 100 dp(density-independent pixles). The “bluetooth” button will be placed above the “first_person_button” in the point cloud App.

2. Add the string of “bluetooth”

Next, go to “strings.xml”. Add one line of code:

/******/

</string>
<string name=”bluetooth”>bluetooth</string>

/******/

This gives the string type of “bluetooth” that can be called in the “activity_jpoint_cloud.xml”

3. Modify the code in point_cloud_activity

Now, go to “PointCloudActivity.java”. Add the following private classes:

/******/

private BluetoothAdapter BA;
private Set<BluetoothDevice>pairedDevices;
private boolean bluetoothFlag = false;

/******/

This declares Bluetooth adapter and store the list of paired Bluetooth devices. The last line is to store the state of Bluetooth(on/off) because the button can be used as a switch. By initial, it is set to off.

Add one line below:

/******/

BA = BluetoothAdapter.getDefaultAdapter();

/******/

This will assign the BluetoothAdapter.

Now go down to public classes, Add the following code:

/********/

public void bluetoothOn(){
if (!BA.isEnabled()) {
Intent turnOn = new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE);
startActivityForResult(turnOn, 0);
Toast.makeText(getApplicationContext(),”Turned on”,Toast.LENGTH_LONG).show();
visible();
}
else
{
Toast.makeText(getApplicationContext(),”Already on”, Toast.LENGTH_LONG).show();
}
}

/********/
This creates a function “bluetoothOn()” to turn on Bluetooth device and toast a message (“Turned on” or “Already on”).
/********/

public void bluetoothOff(){
BA.disable();
Toast.makeText(getApplicationContext(),”Turned off” ,Toast.LENGTH_LONG).show();
}

/********/

similarly, this creates a function “bluetoothOff()” to turn off Bluetooth device and toast a message “Turned off”

Next add the following lines below:

/********/

public  void visible(){
Intent getVisible = new Intent(BluetoothAdapter.ACTION_REQUEST_DISCOVERABLE);
startActivityForResult(getVisible, 0);
}

/********/

This will alter the Bluetooth device into visible state.

Continue adding these lines of code:

/********/

public void list(){
pairedDevices = BA.getBondedDevices();
ArrayList list = new ArrayList();

for(BluetoothDevice bt : pairedDevices)
list.add(bt.getName());
Toast.makeText(getApplicationContext(),”Showing Paired Devices”,Toast.LENGTH_SHORT).show();

}

/********/

This creates an array and add paired devices into list.

Finally, incorporate the main function:

/********/

@Override
public void onClick(View v) {
switch (v.getId()) {
case R.id.first_person_button:
mRenderer.setFirstPersonView();
break;
case R.id.third_person_button:
mRenderer.setThirdPersonView();
break;
case R.id.top_down_button:
mRenderer.setTopDownView();
break;
case R.id.bluetooth_button:
if(bluetoothFlag == false){
bluetoothOn();
bluetoothFlag = true;
}else {
bluetoothOff();
bluetoothFlag = false;
}
default:
Log.w(TAG, “Unrecognized button click.”);
}
}

/********/

This will create the click event handler. Based on different click event,it will switch viewer’s perspective and turn on/off Bluetooth device. In Bluetooth button section, this function will switch adapter between on/off.

IMG_3210

This is what the App will looks like after the modification. Notice the Button shows upon on the screen.

The modified Java Code can be downloaded here

 

Conclusion:

Modifying an existing Android App could be difficult since it requires good knowledge of Java Programming language as well as understand Android Studio platform. In this approach, I first did research and learn the basic Android App development and Java programming on Udacity website. Then I start modifying code in Point Cloud Existing App but unfortunately I crushed the App over and over again. After we got the Project Tango system block diagram from professor Hill, I start researching into Android Bluetooth API and trying to implement existing bluetooth sample code into the Point Cloud activity. Finally I successfully debug and run the modified App but the button itself does not work properly. After trying to debug the code, I found out that I need assistant from someone who develop the point cloud App to let me know where the breakpoints goes since I lost track of the breakpoints when stepping into the code.

This blog post does not show the fully functional Point Cloud App with Bluetooth feature, but can be documented and used as a good resource for individuals or groups from future class to continue developing the Java Point Cloud App.

 

Source Materials:

Spring 2016 Pathfinder: Project Tango Preliminary Research

projecttangologoblack

by:

Peiyuan Xu (Project Manager)

Nicholas Lombardo (Project Assistant)

 

Table of Contents

Introduction:

This spring 2016 Pathfinder project will use Google’s Project Tango as a platform to test and implement SLAM (Simultaneous Localization and Mapping) technology for autonomous vehicles. Project Tango uses computer vision to give device the ability to know the position in the relative 3D space as well as the distance from the surrounding objects to the device itself.

 

Physical Description

tango_specs1

The Tango tablet has a physical dimension of 119.77 x 196.33 x 15.36mm.

The weight is 0.82 lbs (370g)

Tango_hw

On the back of the tablet, it has a fisheye camera placed in the middle with an IR projector on the right and a RGB-IR camera on the left. The IR projector and RGB-IR camera work together with IMU (Inertial Measurement Unit) sensors to perform the features of Motion Tracking, Depth Perception and Area Learning.

 

Key Features

  • Motion Tracking

The IMU is used to give feedback on the device orientation and spatial displacement from a reference point. The Motion Tracking feature is usually achieved by integrating readings from orthogonally placed gyroscopes, magnetometers, and accelerometers.

Learn more about Motion Tracking

  • Depth Perception

Tango’s Depth perception can provide feedback on distances between the device and nearby solid objects. Tango uses an approach called structured light sensing where a pattern of light is cast and the size and shape of the pattern seen by the camera is used to determine the distance.

Learn more about Depth Perception

  • Area Learning

Tango’s Area Learning feature helps with motion tracking by correcting drift and helps with navigation of indoors applications. It uses image processing of visual features in a virtual environment to recognize a previously-visited location.

Learn more about Area Learning

 

Limitations:

  • The IR sensors for depth perception is generally restricted to indoor environments (sun and incandescent IR light can drown out structured light sensing). Therefore, in order to accomplish the requirements for both solar panels implementation (under direct sunshine) and Tango applications (without sunshine), the mission course is changed to be at CSULB campus during night time.

 

  • Range of IR sensors practically limits its use to nearby obstacle detection rather than long-range path creation. The range of the IR sensor is from 0.5 meters to 4 meters based on Project tango’s website.

 

Power Test For Tango Tablet

IMG_3208 IMG_3210 IMG_3215

The purpose of this test is to gain the power specs while running point cloud App, screen on, bluetooth on, WiFi on for worst case. 

The result shows that it takes about 2 hours and 16 minutes to run the battery from fully charged (100%) to 14%. In which case, it dose not meet our level 1 requirement of running for 4 hours. Also it is recommended to use a high-power 2A USB hub to charge the Tango Tablet.

Therefore, in order to meet the level 1 requirement, an extra portable charger (Power output: 5V, 2A) with Micro USB cable will be used to charge Tango tablet continuously during the mission.

Reference:

 

Get started with Project Tango

Notice that Project Tango offers APIs in C and Java, and a SDK for Unity. For developers who are already familiar with Android studio and Java development, the Java API is the one to use. Developers who want to be able to write apps with the Android NDK (Native Development Kit) should use the C API, which enables more flexibility on the native and system level. The Unity SDK is good for game development in 3D virtual environment.

Learn more on API Overview

Tutorials can be found here:

Once you have downloaded the sample code and successfully installed the App. you will see a window like this:

Screenshot_2016-04-07-22-38-12

This is the Point Cloud version of a person standing in the Hallway. On the top left, It states that “Average depth(m): 4.352” which means that person is about 4 meters away from the tablet.

On the right side, there are 4 buttons. The first “Bluetooth” button is the custom button that I made in order to connect the tablet to Arduino through BlueTooth while running the Java Point Cloud App. Below the “Bluetooth” button, there are the three buttons that can switch from different orientation and give different point of view of the Point Cloud data.

After we have done that, now it is important to understand how the system should work between Project tango and Arduino and then eventually implement the software modules on Arxterra.

 

Project Tango System Block Diagram

tango_system
Above is the Project Tango System Block Diagram that provided by Professor Hill. The diagram is  divided into two parts. In the Android Operating system, commands will be sent by Arxterra to WiFi Manager API and decoded in Tango Android OS. Then the main point cloud App will process the coordinates and position information from Sensor Event API and Tango XYZij structure and send feedback telemetry back from WiFi Manager API to Arxterra. In the Arduino part, Java Point Cloud App will send command sequence through bluetooth to UART subsystem decoder and then to Arduino board. Then the microcontroller (Arduino mega) will send serial read/write to command decoder to command handler and then to GPIO to control the Ultrasonic sensors, H-bridge motors and servos. From that, our team develop the software modules and tasks breakdown to accomplish the system design.

 

Software Modules and Task Breakdown

Module 1:   Sending commands to WiFi Manager API (Through “cool term”)

Module 2:  WiFi Manager API sending telemetry out to Arxterra

Module 3:  Tango SDK or NDK (Translate XYZij to Tango)

Module 4:  SensorEvent API (IMU)

Module 5:  BlueTooth API to HC-06 Module to Arduino

Module 6:  Arduino to HC-06 to BlueTooth API

Module 7:  Arduino Command Decoder to Command Handler

Module 8:  Arduino Command Handler to GPIO Output to Motors and Servos Control

Module 9:  GPIO Input to Telemetry Send Packets to UART Receiving Subsystem to Android OS

Module 10  Java App Command Sequence Development

 

More on IMU and Coordinate System

Module 3 requires to understand the IMU (Inertial Measurement Unit) and Project Tango Coordinate Frames. The links below explain the Tango Coordinate System Conventions

Coordinate System Convention

The next link explains Android devices IMU sensors in general that includes Tango Tablets as well

Sensors Overview

Research into PCL and ROS

Module 10 requires knowledge of processing point cloud data. This may be as easy as importing a library and using the defined classes/methods for data manipulation.

The PCL (Point Cloud Library) and ROS (Robot Operating System) are both open sources that offers libraries and tools to analyze point cloud data. Many companies and developers are working on building application with PCL or ROS for Project Tango. Some useful Tips and tutorials can be found here:

How to build applications with PCL for Android

Getting Started with Tango and ROS

PCL Documentation

 

Conclusion

The use of Project Tango tablet is ambitious since it is fairly new and requires extensive knowledge of Android App development and Java Programming. In this semester, our team is able to attempt and finish some of the Project Tango modules, that includes  “BlueTooth API to HC-06 Module to Arduino (Blinking LED and Motors Control)”, “Extract Point Cloud Data from Tango and process with Paraview software”. We will have upcoming blog posts to explain and show the steps of how these modules can be done. We hope the research work we have here can benefit the future groups or individual who wants to continue work on Project Tango.

Source Materials: