This is one of those thought experiments – can I create a neural network that returns Math.Sin()? We don’t need to also build one for Math.Cos() as the two are the same except out of phase by 90 degrees.
Here’s the result:
Trust me I pondered for a few minutes to think how to display the results. Numbers are boring, so I created the above transparent filled polygons.
I was somewhat surprised to read recently from someone who is clearly much more experienced than me that Sin() wouldn’t seem to train for him, leading to inaccurate values. If you’re still not convinced it works accurately, uncomment this bit (in the constructor).
Bitmap b = new(720, 200);
for (int x = 0; x < 720; x++)
{
float y = SinViaAI(x/2)*100+100;
b.SetPixel((int) x, (int) Math.Round(y ), Color.Black);
}
b.Save(@"c:\temp\sine-wave.png", ImageFormat.Png);
If unclear on how the polygon animation how it works, please see my animation post.
For those of you who know how to draw a polygon, please skip to the training.
Visualise the process as follows: draw a dot on a piece of paper (this is the centre), using a compass draw a circle around it. Draw a line from the centre to anywhere on the circle (shown in red). Calculate the angle required based on 360/number of sides in our case 72. Get a protractor and mark a point that is that angle from the line. Draw a line from the centre thru that point all the way to the radius. Repeat this process until you arrive back at the start line. Everywhere it intersected with the circle is a corner. Draw lines between the corners. You have a polygon with equal sides.
I learnt to draw polygons around 10 years of age by accident; and used that knowledge aged around 15 to develop a print-n-plotter driver for an Amstrad dot matrix printer (it allowed you to feed/reverse-feed in very small increments).
/// <summary>
/// Draw a polygon to prove that our AI returns SIN.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void DrawPolygon(int sides)
{
Bitmap polygon = pictureBox1.Image is null ? new(pictureBox1.Width, pictureBox1.Height) : new(pictureBox1.Image);
using Graphics g = Graphics.FromImage(polygon);
g.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighQuality;
g.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighQuality;
List<PointF> p = new();
Random random = new();
for (float angle = 0; angle < 360; angle += 360 / (float)sides)
{
float r = 200-sides*10;
p.Add(new PointF(pictureBox1.Width / 2 + r * SinViaAI((angle+offs)%360),
pictureBox1.Height / 2 + r * CosViaAI((angle+offs)%360)));
}
using SolidBrush brush = new(Color.FromArgb(40, random.Next(255), random.Next(255), random.Next(255)));
g.FillPolygon(brush, p.ToArray());
p.Add(p.ToArray()[0]);
g.DrawLines(Pens.White, p.ToArray());
pictureBox1.Image?.Dispose();
pictureBox1.Image = polygon;
}
That logic uses a SinViaAI() method declared as below, notice the FeedForward(), and Cosine being 90 degrees out of phase Sine.
/// <summary>
/// Returns Math.Sin(), using AI!
/// </summary>
/// <param name="angle">Angle (0..1)</param>
/// <returns></returns>
private float SinViaAI(float angle)
{
if (angle < 0 || angle >= 360) throw new ArgumentOutOfRangeException(nameof(angle),"0<=x<360");
return neuralNetwork.FeedForward(new float[]{ angle / 360F})[0];
}
/// <summary>
/// Returns Math.Cos(), using AI.
/// Cos(a) == Sin(a+90).
/// </summary>
/// <param name="angle">Angle (0..1)</param>
/// <returns></returns>
private float CosViaAI(float angle)
{
return SinViaAI((angle + 90F) % 360F);
}
Training
The configuration was somewhat trial and error. Please feel free to experiment. It seems you can’t connect the input to the output as you can for lots of linear problems. Adding a small number of hidden neurons didn’t seem to train. It worked with this, 360 hidden neurons seems like a lot. If you can reliably train in less, please comment below.
INPUT: angle in degrees / 360 OUTPUT: Math.Sin()
int[] layers = new int[6] { 1 /* INPUT: angle/360 */,
90, 90, 90, 90,
1 /* OUTPUT: Math.Sin(angle) */
};
Rather than compute Sine repeatedly, we cache the training data
float[] angleDiv360toBetween0and1 = new float[360];
float[] sinOutputForAngle = new float[360];
// do this once, and cache as Sin() is slow.
for (int n = 0; n < 360; n++)
{
angleDiv360toBetween0and1[n] = n / 360F;
sinOutputForAngle[n] = (float)Math.Sin(DegreesInRadians(n));
}
We then train via back-propagation, the basic idea is trying up to 50,000 times until it is correct.
// train the AI up to 50k times, exiting if we are getting correct answers.
// note: it could fail even with 50k, simply because it picks bad initial weights and biases.
for (int i = 0; i < 50000; i++)
{
for (int n = 0; n < 360; n++)
{
float[] inputs = new float[] { angleDiv360toBetween0and1[n] };
neuralNetwork.BackPropagate(inputs, new float[] { sinOutputForAngle[n] });
}
}
I wrote it before I made a simple framework for training, so it’s a little basic.
We need to exit before 50,000 if we don’t want to wait endlessly. The approach I often take is to wait a predefined number of epochs (times around the training loop), and then when we reach that point check whether all the training points match the training data. If they pass, we break out of the loop (saving endless waiting).
// by this point we *may* have done enough training...
// we check the output, and if it's accurate, we exit. We don't check before 5k,
// because it would slow training down for little gain.
if (i > 5000)
{
trained = true;
// test the result for all permutations, and if all are good, we're trained.
for (int n = 0; n < 360; n++)
{
float[] inputs = new float[] { angleDiv360toBetween0and1[n] };
float expectedResult = sinOutputForAngle[n];
float prediction = neuralNetwork.FeedForward(inputs)[0];
if (Math.Abs(expectedResult - prediction) > 0.01F) // close enough (accuracy)
{
// enable this to see how close it gets...
Debug.WriteLine($"{n} AI:{prediction:0.000} Expected: {expectedResult:0.000}");
trained = false; // wrong answer
break;
}
}
if (trained)
{
Text = "Sin Wave - TRAINED."; // we know it passed training
neuralNetwork.Save(c_aiModelFilePath);
break;
}
}
if (i % 1000 == 0) // indicator of progress, every 1000
{
Text = $"Sin Wave - TRAINING. GENERATION {i}";
Application.DoEvents();
}
I have provided the weighting/bias sinWave.AI file, so you can see it without training it yourself.
The source code for this is on GitHub.
There are lots of “other” things one can do, from making the output passthrough (no TanH), to using other activation functions (LeakyRELU and others are more popular it seems). What’s the minimum number of neurons you can train it in?
This post started life as a “can it be done?”, and the answer is yes. If you found this useful, please let me know in the comments.
