In this presentation, we will explore the technologies that go into making successful autonomous functions in robots, dive into several examples of automation in StuyPulse’s past robots, and finally, the process of designing and building an automated system.

What is automation?

Automation is a set of techniques that allow a system to operate with minimal human input.

Why automate?

In FRC, there are many things the driver must do: find and pick up game pieces, drive the robot to the scoring location, align and score the game piece(s). All of these actions may be done completely manually, however, that would be much more time consuming than having the robot be able to complete these tasks by itself. Of course, it would be extremely challenging to build a robot capable of full autonomy, however, semi-autonomous robot functions can be used to aid drivers in doing tasks requiring precision and accuracy such as placing a cone on a peg or aligning the robot towards a shooting goal. Many teams have used automation to speed up cycle times and aid robots into becoming more accurate.

4 Pillars of Autonomy

Let’s dive into the world of autonomy which I have broken up into 4 sections: information gathering, decision making, control, and navigation.

Information Gathering

The robot by itself is completely blind. It has no way to understand the state of both itself and the world around it. How can we fix that? We can use information gathering through sensors and processing the data these sensors give you.

Information gathering is integral to automation. With the ability to read information from the outside world, the robot can understand what it’s doing instead of blindly following whatever input it receives from an external source such as a human or timer system. Instead of just following instructions along the lines of “run motors 1, 2, 3, and 4 at 6 volts for 3 seconds,” the robot can now receive instructions like “run motors 1, 2, 3, and 4 until they have made the robot reach a distance of 5 meters.” Information gathering allows us to have better, more reproducible control over the robot, rather than just guessing it should run for this long and hoping it gets to the right position.

Sensors

Encoders (Absolute / Relative)

Encoders are sensors which detect rotary or linear displacement.

Distance Sensors (Ultrasonic, IR, LIDAR)

Distance sensors measure the distance between the sensor and an object within its range.

Ultrasonic sensors use ultrasonic waves, IR sensors use beams of infrared light, and LIDAR utilizes laser.

Proximity Sensors (Microswitch, Break Beam, IR)

Proximity sensors measure whether something is present or not.

Color Sensors

Color sensors measure the color of an object within its range. It may spit out values in either RGB or HSV.

IMUs (Inertial Measurement Unit)

Gyroscopes measure the orientation. Accelerometers measure the acceleration the sensor experiences. An IMU is a combination of both of these sensors.

Cameras

Cameras continuously capture a snapshot of the outside world, allowing for the robot to “see.” Of course this image data is completely useless as raw data, and further processing is necessary to derive more specific and applicable information from the video stream.

Computer Vision

Computer vision is the process of gathering, processing, and analyzing digital images to extract relevant data from the outside world.

There are many ways to derive information from an image and many things that can be extracted.

A couple examples of what computer vision can do:

• Color detection / isolation
• Object detection / localization
• Visual odometry / localization

In FRC, computer vision was previously focused on color detection and target tracking with retroreflective targets. Color detection could be utilized to track game pieces and retroreflective target tracking could be used to move towards and aim at scoring targets or human player stations.

FRC has moved onto using AprilTags on the field, which are unique artificial markers placed in known locations around the field. With this knowledge, a vision system may detect and isolate these tags and calculate the robot’s absolute position on the playing field. AprilTags provide a source of truth for position on the field, making full field localization much more accurate and accessible to teams.

There are many tools that make computer vision plug and play on robots, with a little bit of tuning. Solutions such as the open source PhotonVision, which requires a bit more setup work, or the Limelight ecosystem, which requires you to purchase a $400 product, make vision as simple as possible.

In 2023, StuyPulse’s robot Jim utilized 2 Limelights which were running AprilTag detection pipelines. These pipelines allowed for us to track where the robot is on the field whenever we were within a certain range of the AprilTags. Through our testing, the vision data we received became increasingly noisy and inaccurate. This led us to only utilize vision data within a certain range, and to decrease the weight of the vision data on the calculated pose of the robot outside that range, or cut it off completely.

This pose data allowed us to do pose alignment, where we have known poses for each scoring position and we’re able to consistently and accurately align to each scoring position.

Investment into vision systems is a good project to undertake during the off-season.

Sensor Fusion

All sources of data have a level of uncertainty or noise, small differences between readings due to mechanical or electrical inefficiencies. Sensor fusion is the combination of multiple sources of data together which results in data with less uncertainty and higher accuracy.

Some sources of data may also have differing rates at which it returns data, which means too-old data may be being used to control the system.

You may also apply a weight to each source of data to control how much you want each source to impact the output data.

For example, an encoder might return data every 20ms but a vision system may only return data every 100ms. If you only used the vision system, old data may lead to your system overcompensating since it thinks it’s farther away than it is to the target. However, solely using encoder data can be noisy and extremely uncertain, since it does not directly read what’s in the outside world. However, if you fuse these two sources together, with the encoder supplementing information into the system between the time it takes for the vision system to return data, and the vision data correcting the encoder data, you have a more accurate output data which can be used to control your other systems.

Filtering

A filter is a mapping of inputs to outputs. Filters can be applied to smooth out noisy data, limit the rate at which a signal can change, or filter out extremes. Filtering may add delay, or phase lag, to the signal since it can only work on previous values.

A variety of filters exist:

• Moving average filter: Take the average of a range of previous data.
• Median filter: Take the median of a range of previous data. Good at taking out sudden spikes in the signal.
• High pass filter: Filters out signals above a cutoff frequency.
• Low pass filter: Filters out signals below a cutoff frequency.
• Slew rate limit: Primitive motion profile on velocity or voltage signals. Caps maximum rate of change.
• Debouncer: Eliminates unwanted sudden changes in binary values. Requires a change in binary value of a signal to be more than a certain period of time before the signal output value is changed.

Decision Making

Now that the robot has means of measuring its own state and the world around it, you can build out logical trees which act as a decision making mechanism in the robot. They’ll use the data gathered from sensors to determine the state of the robot and whether certain actions should be triggered. These actions can range from a simple change in LED indicator lights to full blown automated game piece indexing or scoring.

Sensors in Decision Making

Sensor data can be binary, or they can have a range of values. Parsing through the data sensors provide you into a range of values is important in later steps. These values sensors provide you can be triggers for various mechanism states.

For example: A color sensor might send you data in the form of RGB values detected by the sensor. You must be able to compare these values to a set of acceptable values which means one of several things, such as whether red, blue, or no balls are detected.

State Machines

A robot or mechanism may be in a variety of states, such as whether or not the arm of your robot is extended or retracted, or if the shooter is ready to shoot. These states make up a state machine, or a finite state machine, where there is a set number of states a system can be in. These states may be triggered by sensor readings such as whether a limit switch has been pressed, a ball has been acquired into the robot, or whether the human driver tells the robot to go to a certain state.

Certain states can trigger the robot to complete certain actions and/or trigger states in other mechanisms of the robot to change.

Logical Flow

With the ability for mechanisms to operate based on states, complex logic trees can be constructed to trigger these states. Logical flow diagrams can be used to visualize the logical flow of a system and aid in implementation.

Control

Control Theory

Use of algorithms to influence a system to reach a desired goal.

Motion Profiling

Control algorithm that defines movement as a series of steps or smaller motions. Can be used to ramp a mechanism’s change in velocity or acceleration and cap it.

Feedforward

Also known as open-loop control, these control algorithms are blind to the system’s current state and continuously feed output into the system. Common usages of feedforward control in FRC is to continuously apply enough voltage to overcome static friction or gravity.

Feedback

Also known as closed-loop control, these control algorithms utilize knowledge of the system’s current state and the desired state to drive the system closer to the desired state. The output of the controller is changed as the state of they system changes.

System Identification

the process of empirically determining feedforward and feedback gains through collecting and analyzing data from a series of tests

Navigation

Now that you’re able to track the state of the robot, decide whether or not the robot is in its ideal state, and your robot can achieve its target state using control loops, it’s time for the final step, getting your robot to autonomously drive around.

Odometry

Odometry is the use of motion sensors such as drivetrain encoders and IMUs to estimate the change in position of the robot over time.

The most common odometry technique is dead reckoning, a process which uses knowledge of the previous position to calculate the current position by incorporating estimates of speed (encoders), heading (gyroscope), and elapsed time.

Dead reckoning is susceptible to error over time, since it only uses data relative to the robot, with no source of data which knows the absolute position of the robot. Things like the wheels skidding on the ground or going on a tilted path are not able to be compensated for since the wheels have traveled that much distance, however the robot has not.

Visual Odometry

Visual odometry utilizes vision data to calculate the position of the robot in a space. This can be through the use of feature tracking such as artificial markers like AprilTags or natural features such as unique building features or objects in the environment.

Pose Estimation

As previously mentioned, sensor fusion combines multiple data sources to output data which is more accurate. Pose estimation is the use of a combination of multiple sources of localization data to approximate the position of the robot in a space. Typically in FRC, this means the combination of odometry data from dead reckoning with vision measurements made by your computer vision subsystem.

Other custom means such as environment mapping, combining spatial data gathered by a LIDAR system with odometry and robotic exploration, may gain prevalence in the future.

Pose estimation is extremely powerful in that with enough tuning, the localization data output from the pose estimator can be within centimeter precision. Pose estimation can correct for odometry shift

Configuration Space

Collision avoidance is extremely important in attempting to control a robot. Physical limits such as extension limits or intersections with other parts of the robot will limit the range the mechanism can reach.

Configurations are individual states the mechanism can be in. The configuration space is the set of configurations which are physically achievable by the mechanism.

The simplest configuration space would be a 1DOF configuration space, including elevators, telescoping arms, or turrets.

Mechanisms with a 2DOF (or even N-D) configuration space include drivetrains, 2 jointed arms, or elevator + arm combos. You can represent these configurations in an (x, y, …) format.

Graphical representations of these configuration spaces will include blocked out areas which are obstacles or ranges of configurations which are invalid since the mechanism cannot or should not reach them.

Motion Planning

We want the mechanism to reach point A to point B within the configuration space. The path, consisting of the endpoints A and B, along with any intermediate points, is known as a trajectory. We can use either a visual interface which will allow us to see and edit points of the trajectory, or plain coordinates in code.

Although these techniques are primarily used in running the drivetrain in FRC, it is also possible to get other mechanisms such as multi-jointed arms to use motion planning, albeit requiring some custom code for it to work.

Path Following

Now that we have a trajectory which the robot should follow, we need to somehow convert that trajectory into a stream of inputs to the mechanism which will follow to get to the desired points.

Designing Your Own

Now that we’ve seen the components that go into autonomous robot functions and routines, let’s talk about how you can design and integrate automation into your robot.

1. Identify what can be automated.
2. Plan for how you want it to be automated.
3. Ensure that the robot will have the necessary sensors and components to allow the automation to exist and work.
4. If you’re automating something with many states and many sensors which influence what state the mechanism is in, write out a logical flow diagram.
5. Implement the automation, using control algorithms as necessary.
6. Use system identification to gather feedback and feedforward constants.
7. Run and test your automation.
8. If necessary, revise your code if unexpected behavior is present. You may encounter edge cases through testing. Constants may need to be manually tuned.

The slideshow for this presentation can be found here. This presentation was given at StuySplash 2023.