What’s the secret to training the AI bees?
Overview
The primary reason the bees improve (avoid, and collect) is through mutation – we discard the worst-performing bees, leaving the bees that perform best. Whilst nature has “natural selection” presumably also via mutation, the underlying mechanics as implemented are too inefficient for the real world for reasons.
A viable mutation approach is very much dependent on being able to identify best-performing bees, primarily encouraging the behaviours we seek – we’ll cover that too in detail.
Apparently, a bee has around 1,000,000 neurons squeezed into its tiny head – my AI bees have just 62 because I don’t think 1mn is actually going to make it work any better, and the bees will be the speed of snails. Bees live an extraordinarily complex life, whereas the AI world is simplistic; to start with it’s a 2D world, without predators (hornets, birds etc), and no wind/rain. Bees have had more than 100+ million years of evolution to organise their brains and we don’t have that long to train AI.
A perceptron network provides the “smarts” with 60 sensor inputs (for eyes, spread across 240 degrees) directly connected to 2 outputs (one for each wing).
Bees can rotate and drift to move. I cannot call this a precise “simulation”, as that was never the goal, but the AI is able to learn to do both.
Vision
Living in 2D means the vision doesn’t need to be complex. Increasing the resolution won’t hugely change the behaviour as we’ll cover.
The output from my “vision” system differs subtly from my other applications. This is one of those “try-it-and-see” types of experiments that seem to work.
Each INPUT neuron provides a distance from the bee to the objects to detect (wall, tree, bee, flower) in a specific direction. The diagram shows a mere 17 sensors, there are in fact 60 sensors, drawing all will not improve the diagram.
Where the approach differs from my usual modus operandi is that:
- we use a positive 0…1 if the object is a flower (0.1 indicates close, 1 = further away but visible, 0=nothing detected)
- we invert the value (0…-1) for all other objects and reverse the value such that the closer gives 1 and the further away gives 0.1.
The point of positive/negative is to encourage the bee to head towards flowers whilst avoiding collisions with other objects. The purpose of “reversing the value” is so that values for closer objects encourage the bee to stay away (more value = more avoidance necessary).
It seems to work because TANH is happy with +/-1.
The output of the neural network is amplified to improve the speed of the bees. TANH as an activation function has a limit of 1, and for a better experience, we need something a little bigger.
Real bees have interesting compound eyes, but still logically separated into two thus providing stereoscopic vision – providing depth perception using 2 almost identical images (in a forward direction), that difference providing additional queues on depth. We could absolutely do that too, by having 2 sensors separated by the width of the bee that overlaps in the central region and changing the computed “depth” (binocular disparity) into the raw pixel colours.
I have indeed implemented an alternative “stereo” vision with 2 “eyes” – set it to true.
/// <summary> /// True - it uses stereo vision, and struggles to achieve anything good. /// False - it uses mono vision (LIDAR style), and works well. /// </summary> internal const bool c_useStereoVision = false; // default: false
But it doesn’t work well at all…
And this is where understanding and complexity have other ideas.
The central vision comprises two images offset slightly, known as binocular vision. At the sides, you have peripheral vision and it is known to be poor for depth perception, but great to spot predators etc.
Or so that’s the long thought belief. But in fact, we are perfectly able to judge closeness even with a single eye – a bit of trivia, I don’t have binocular vision yet can still drive a car. This site explains the visual queues used by monocular vision.
You instantly identify the image below as a train track. Being smart you understand the rails are parallel (going into the distance not upwards), otherwise, the train will struggle given the axles don’t reduce in length as it travels down the track! You also know that all the sleepers under the track are the same length and just look shorter due to the vanishing points.
That’s how amazing your brain is, but before you feel too good, remember all living creatures are no less able.
Having 100 compound eyes is too much for a simple AI. It pretends by seeing from both eyes as if the head was transparent, I know that would be weird although no more than this. Maybe I’ll revisit it, or you can create your own replacement class, and let me know how well it works.
The bee will see the number of pixels of colour indicating a tree increase in number the closer the bee is. The same for the flower. Understanding that relationship is easy to take for granted.
Experiments with vision have piqued my further interest, let’s not get bogged down any further on the complexities here.
To simplify things for the AI, we add clarity to the environment, redrawing it with the things that matter. Below is what the bee sees – drawn as a silhouette from the real image.
Is it slightly cheating? It depends on what the objective was, to begin with. This isn’t designed to be an accurate bee simulator, it’s about highlighting the different challenges.
Mapping Vision To Movement
Controlling the bees started off with the control approach (see the earlier posting about cars and tanks), with the input driving 2 throttles. However, tanks do not usually move sideways, probably something about weighing 50 tons and having caterpillar tracks. I have thus added a rudimentary sideways drift based on the differential of wing input.
When scientists made the stupid statement a bee shouldn’t be able to fly, it was the result of ignorance. With high-speed cameras, it soon becomes pretty obvious how thrust is generated. Trying to mimic all the subtleties doesn’t improve the AI, so I opted for a more common sense approach that does not factor in the position or angle of the wing.
If both wings beat the same amount, the bee goes forwards. If one wing beats more in my physics, it turns to the side beating less. I am sure a real bee rotates the leading wing in the direction (trailing wing back) it wants to turn to effect a force backwards and sideways thus turning it. The physics also supports lateral movement. To avoid overly complex logic, in our bees this occurs when one wing beats forward, and the other backward. You might reasonably argue in reality that would rotate the bee…
i.e the logic when searching for nectar is:
/// <summary>
/// AI logic to move forward avoiding walls / trees / bees.
/// </summary>
private void SearchForNectarAvoidingWallsAndTrees()
{
// input is the distance sensors of how soon we'll impact
double[] neuralNetworkInput = Config.VisionSystem.VisionSensorOutput(id, AngleBeeIsPointingInDegrees, Location);
// provide the neural with vision and let it decide what to do with the bee
double[] outputFromNeuralNetwork = NeuralNetwork.s_networks[id].FeedForward(neuralNetworkInput); // process inputs
ApplyPhysics(leftWingFlapRate: (outputFromNeuralNetwork[0] * Config.OutputModulation[0]).Clamp(-0.1, 1.5),
rightWingFlapRate: (outputFromNeuralNetwork[1] * Config.OutputModulation[1]).Clamp(-0.1, 1.5));
// accumulate how far it has moved
DistanceTravelled += MathUtils.DistanceBetweenTwoPoints(Location, LastLocation);
UpdateRadar();
}
The “physics” in code form is as follows:
/// <summary>
/// Moves the bee using speed at given angle, with the ability to drift sideways where
/// there is a large enough difference in flap speed.
/// </summary>
/// <param name="beeInputState"></param>
internal void ApplyPhysics(double leftWingFlapRate, double rightWingFlapRate)
{
// based on which wing is beat harder, we steer in the direction of the slower
AngleBeeIsPointingInDegrees += 30 * (rightWingFlapRate - leftWingFlapRate) / 4;
// if both are "1" we move top speed, anything else is less than.
Speed = (leftWingFlapRate + rightWingFlapRate) / 2;
if (Speed == 0) Speed = 0.5;
// it'll work even if we violate this, but let's keep it clean 0..359.999 degrees.
if (AngleBeeIsPointingInDegrees < 0) AngleBeeIsPointingInDegrees += 360;
if (AngleBeeIsPointingInDegrees >= 360) AngleBeeIsPointingInDegrees -= 360;
// move the bee using basic sin/cos math -> x = r cos(theta), y = r x sin(theta)
// in this instance "r" is the speed output, theta is the angle of the bee.
double angleBeeIsPointingInRadians = MathUtils.DegreesInRadians(AngleBeeIsPointingInDegrees);
Location.X += (float)(Math.Cos(angleBeeIsPointingInRadians) * Speed);
Location.Y += (float)(Math.Sin(angleBeeIsPointingInRadians) * Speed);
// bees can fly sideways, if enabled (rather than rotate)
if (Config.BeeCanDriftSideways)
{
if (leftWingFlapRate > 0 && rightWingFlapRate <= 0)
{
angleBeeIsPointingInRadians -= MathUtils.DegreesInRadians(90);
Location.X += (float)(Math.Cos(angleBeeIsPointingInRadians) * Speed * 5);
Location.Y += (float)(Math.Sin(angleBeeIsPointingInRadians) * Speed * 5);
}
else
if (leftWingFlapRate <= 0 && rightWingFlapRate > 0)
{
angleBeeIsPointingInRadians += MathUtils.DegreesInRadians(90);
Location.X += (float)(Math.Cos(angleBeeIsPointingInRadians) * Speed * 5);
Location.Y += (float)(Math.Sin(angleBeeIsPointingInRadians) * Speed * 5);
}
}
}
Note the “speed = 0.5”, ensuring the bee moves. I strongly dislike doing it, but without it, one risks the bees not moving in the hive and impeding ones that might be successful.
Scoring
Using “scoring” is a strange but effective way to train.
We need to come up with a way to first incentivise the bees to leave the hive, by rewarding the 1000 points for leaving plus extra points the further they have flown.
Since the goal is nectar collection, it makes sense to reward the bee for each flower they visit, we give them a handsome 10,000 per flower.
If they successfully return to the hive to sleep, they got lucky or have some collision avoidance, we reward them a bonus of 1000.
Lastly, we don’t want bees lounging around the hive all day, so for that, they get a punishment in the form of a 0 score.
The logic in code is as follows:
// The further the bee flies the more points. // The goal is nectar, so more flowers visited the better the bee, float fitness = DistanceTravelled + nectarCollected * 10000; // We need to encourage them to leave the hive, so bonus 1000 for that. if (nectarCollected > 0 || Location.X > 111) fitness += 1000; // Extra points for returning home to sleep. if (currentTaskOfBee == BeeTask.Sleep) fitness += 1000; // They get no points for being near where they started. if ((currentTaskOfBee == BeeTask.CollectNectar) && MathUtils.DistanceBetweenTwoPoints(StartLocation, Location) < 40) fitness = 0; NeuralNetwork.s_networks[id].Fitness = fitness;
Replicating Tasks
Our focus is on them leaving the hive, seeking nectar and returning with it, making our tasks:
Wake up > collect nectar > return to the hive > sleep
Collecting the nectar we use AI. For returning to the hive, we cheat. Yes, no point in lying, given the code is on GitHub!
The bees are provided one of two locations depending on where they are – the red crosses below. If the bees (see top left three) are instructed to return home, they cannot go directly to the hive. Even bees cannot violate the laws of physics, and given these are new walls without holes, they are blocked. To “help” them, we direct them to the “X” on the right until they are just past the wall. From there they target the other cross on the left.
If the bees are not at the top left, they aren’t obstructed, we send them to the “X” near the hive.
Upon arriving they are directed to the next available bed.
Alas, routing isn’t quite that simple thanks to the presence of trees…
The circled bee is not permitted to fly through trees. Hit the tree, they get squished. We have a conflict that we overcome with a somewhat basic algorithm. We get the bee to check the radar. If the centre three sensors don’t indicate an obstruction close by, we move towards the “X”. If there is an obstruction we revert to the AI. Why does that work? Because the bee is pointing at the angle we want it to go, the AI knows to avoid obstacles and takes corrective action, eventually having avoided, the path is clear and it can head towards the “X”. It isn’t perfect, but works quite well.
What the bee “sees”.
And in action…
Oops. No, that wasn’t what it was meant to do!!!! What part of “do not hit trees” was unclear? Oh, all of it. I guess that bee was too excitable for this task.
The bees in the following clip “understand” what was expected. The long red line on their radar indicates the way-point they are heading to – two of them avoid the tree.
It’s an incredibly simple approach, hence this simple code.
We thus track a “state” (currentTaskOfBee) that determines what we want the bee to do (CollectNectar > ReturnToHive > ReturnToBed > OrientAngle > Sleep)
/// <summary>
/// Read the sensors, provide to the "brains" (neural network) and take action based on
/// the output.
/// However, behaviour depends on the current "task"
/// </summary>
private void ApplyAIoutputAndMoveTheBee()
{
LastAngleBeeIsPointingInDegrees = AngleBeeIsPointingInDegrees;
// retain last location
LastLocation = new PointF(Location.X, Location.Y);
PointF DesiredPosition;
switch (currentTaskOfBee)
{
// the main task, collecting nectar.
case BeeTask.CollectNectar:
SearchForNectarAvoidingWallsAndTrees();
return;
// bee is somewhere in the playground, it needs to head towards hive entrance
case BeeTask.ReturnToHive:
// if it is above the diagonal line, aim the bee towards centre of screen
if (Location.X < 110 && Location.Y < -0.4762 * Location.X + 240)
DesiredPosition = new PointF(300, 260);
else
DesiredPosition = new PointF(85, 260);
break;
// bee is in hive, now needs to target the bed.
case BeeTask.ReturnToBed:
DesiredPosition = HiveManager.GetNextFreeBed();
break;
// bee is in bed, facing wrong direction - needs to face upwards
case BeeTask.OrientAngle:
RotateBeeToFaceUpwards();
return;
default:
throw new Exception("illegal unhandled task"); // what did you want the bee to do?
}
float angleInDegrees = (float)MathUtils.RadiansInDegrees((float)Math.Atan2(DesiredPosition.Y - Location.Y, DesiredPosition.X - Location.X));
#if showDebugForTargetting // enable to see where each bee is targetting
AngleRequested = angleInDegrees;
#endif
float deltaAngle = Math.Abs(angleInDegrees - (float)AngleBeeIsPointingInDegrees).Clamp(0, 30); // max bee can proceed
// quickest way to get from current angle to new angle turning the optimal direction
float angleInOptimalDirection = ((angleInDegrees - (float)AngleBeeIsPointingInDegrees + 540f) % 360) - 180f;
// limit max of 30 degrees
AngleBeeIsPointingInDegrees = MathUtils.Clamp360((float)AngleBeeIsPointingInDegrees + deltaAngle * Math.Sign(angleInOptimalDirection));
double[] neuralNetworkInput = Config.VisionSystem.VisionSensorOutput(id, AngleBeeIsPointingInDegrees, Location); /// input is the distance sensors of how soon we'll impact
if (currentTaskOfBee == BeeTask.ReturnToHive)
{
// nothing ahead blocking in first few sensors
if ((neuralNetworkInput[Config.SamplePoints / 2 - 1] == 0 || neuralNetworkInput[Config.SamplePoints / 2 - 1] < -0.3) &&
(neuralNetworkInput[Config.SamplePoints / 2] == 0 || neuralNetworkInput[Config.SamplePoints / 2] < -0.3) &&
(neuralNetworkInput[Config.SamplePoints / 2 + 1] == 0 || neuralNetworkInput[Config.SamplePoints / 2 + 1] < -0.3))
{
Speed = (MathUtils.DistanceBetweenTwoPoints(Location, DesiredPosition) / 10).Clamp(0, 3);
}
else
{
SearchForNectarAvoidingWallsAndTrees();
return;
}
}
else
{
Speed = 1;
}
DistanceTravelled += MathUtils.DistanceBetweenTwoPoints(Location, LastLocation);
double angleBeeIsPointingInRadians = MathUtils.DegreesInRadians(AngleBeeIsPointingInDegrees);
Location.X += (float)(Math.Cos(angleBeeIsPointingInRadians) * Speed);
Location.Y += (float)(Math.Sin(angleBeeIsPointingInRadians) * Speed);
UpdateRadar();
}
Atan2 is the magic that computes an angle from (X, Y) offset that is used a lot in my Missile posts.
100% nectar collected and returned? Nope
Why does the red needle not move all the way to the right?
You probably thought mutation leads to the bees improving each round. It does, but you need to remember that we are always arbitrarily mutating 50% of the bees. We could absolutely use the approaches I used in other posts, where we move the “mutation percentage” needle lower, preserving a higher percentage of bees.
Even with one mutated bee, it will rarely get 100%. The reason is that a mutation may lead to a better or worse bee, it is absolutely not guaranteed to be better. At least 1 is therefore going to have the potential to fail.
Bees can also “obstruct” each other. It is necessary to remove a bee that doesn’t move, esp. if it’s on entry/exit from the hive because you get a jam. The downside is a “good” bee can be inadvertently taken out of the game early because it cannot move due to being obstructed by another bee. Making some kind of engine to place blame and choose, is tricky, so I don’t.
We’ll discuss other failings, soon enough, but it’s important as a takeaway that unless we copy the best AI bee to all 24 bees, and skip mutation we’re going to have failures. If you skip mutation, it won’t improve further. This is where one concedes when it is trained sufficiently and sticking with that best-trained bee.
Where did you go?
The path taken by each bee that hasn’t been eliminated can be turned on/off using the [T] key. Every so often it samples the position; the frequency of doing so is limited such that drawing the lines for every bee doesn’t take too long.
That hopefully covers what the GitHub-hosted application does, and the basics. The rest of this post discusses the challenge of moving from a dumb perception AI to something more akin to a bee.
