Merge pull request #16 from php-ai/develop

Simple Neural Network with MultilayerPerceptron and Backpropagation

CHANGELOG.md

@@ -3,9 +3,11 @@ CHANGELOG
 
 This changelog references the relevant changes done in PHP-ML library.
 
-* 0.1.3 (in plan/progress)
-    * SSE, SSTo, SSR [Regression] - sum of the squared
-    * 
+* 0.2.1 (in plan/progress)
+    * feature [Regression] - SSE, SSTo, SSR  - sum of the squared
+
+* 0.2.0 (2016-08-14)
+    * feature [NeuralNetwork] - MultilayerPerceptron and Backpropagation training 
 
 * 0.1.2 (2016-07-24)
     * feature [Dataset] - FilesDataset - load dataset from files (folder names as targets)

CONTRIBUTING.md

@@ -0,0 +1,43 @@
+# Contributing to PHP-ML
+
+PHP-ML is an open source project. If you'd like to contribute, please read the following text. Before I can merge your 
+Pull-Request here are some guidelines that you need to follow. These guidelines exist not to annoy you, but to keep the 
+code base clean, unified and future proof.
+
+## Branch
+
+You should only open pull requests against the develop branch.
+
+## Unit-Tests
+
+Please try to add a test for your pull-request. You can run the unit-tests by calling:
+
+```
+bin/phpunit
+```
+
+## Travis
+
+GitHub automatically run your pull request through Travis CI against PHP 7. 
+If you break the tests, I cannot merge your code, so please make sure that your code is working 
+before opening up a Pull-Request.
+
+## Merge
+
+Please allow me time to review your pull requests. I will give my best to review everything as fast as possible, but cannot always live up to my own expectations.
+
+## Coding Standards
+
+When contributing code to PHP-ML, you must follow its coding standards. To make a long story short, here is the golden tool:
+
+```
+tools/php-cs-fixer.sh
+```
+
+This script run PHP Coding Standards Fixer with `--level=symfony` param.
+
+More about PHP-CS-Fixer: [http://cs.sensiolabs.org/](http://cs.sensiolabs.org/)
+
+---
+
+Thank you very much again for your contribution!

README.md

@@ -10,7 +10,7 @@
 
 ![PHP-ML - Machine Learning library for PHP](docs/assets/php-ml-logo.png)
 
-Fresh approach to Machine Learning in PHP. Algorithms, Cross Validation, Preprocessing, Feature Extraction and much more in one library.
+Fresh approach to Machine Learning in PHP. Algorithms, Cross Validation, Neural Network, Preprocessing, Feature Extraction and much more in one library.
 
 PHP-ML requires PHP >= 7.0.
 
@@ -62,6 +62,9 @@ Example scripts are available in a separate repository [php-ai/php-ml-examples](
     * [Classification Report](http://php-ml.readthedocs.io/en/latest/machine-learning/metric/classification-report/)
 * Workflow
     * [Pipeline](http://php-ml.readthedocs.io/en/latest/machine-learning/workflow/pipeline)
+* Neural Network
+    * [Multilayer Perceptron](http://php-ml.readthedocs.io/en/latest/machine-learning/neural-network/multilayer-perceptron/)
+    * [Backpropagation training](http://php-ml.readthedocs.io/en/latest/machine-learning/neural-network/backpropagation/)
 * Cross Validation
     * [Random Split](http://php-ml.readthedocs.io/en/latest/machine-learning/cross-validation/random-split/)
     * [Stratified Random Split](http://php-ml.readthedocs.io/en/latest/machine-learning/cross-validation/stratified-random-split/)
@@ -84,17 +87,12 @@ Example scripts are available in a separate repository [php-ai/php-ml-examples](
     * [Matrix](http://php-ml.readthedocs.io/en/latest/math/matrix/)
     * [Statistic](http://php-ml.readthedocs.io/en/latest/math/statistic/)
     
-
 ## Contribute
 
 - Issue Tracker: github.com/php-ai/php-ml/issues
 - Source Code: github.com/php-ai/php-ml
 
-After installation, you can launch the test suite in project root directory (you will need to install dev requirements with Composer)
-
-```
-bin/phpunit
-```
+You can find more about contributing in [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## License
 

composer.json

@@ -3,7 +3,7 @@
     "type": "library",
     "description": "PHP-ML - Machine Learning library for PHP",
     "license": "MIT",
-    "keywords": ["machine learning","pattern recognition","computational learning theory","artificial intelligence"],
+    "keywords": ["machine learning","pattern recognition","neural network","computational learning theory","artificial intelligence"],
     "homepage": "https://github.com/php-ai/php-ml",
     "authors": [
         {

docs/index.md

@@ -62,6 +62,9 @@ Example scripts are available in a separate repository [php-ai/php-ml-examples](
     * [Classification Report](machine-learning/metric/classification-report/)
 * Workflow
     * [Pipeline](machine-learning/workflow/pipeline)
+* Neural Network
+    * [Multilayer Perceptron](machine-learning/neural-network/multilayer-perceptron/)
+    * [Backpropagation training](machine-learning/neural-network/backpropagation/)
 * Cross Validation
     * [Random Split](machine-learning/cross-validation/random-split/)
     * [Stratified Random Split](machine-learning/cross-validation/stratified-random-split/)
@@ -90,11 +93,7 @@ Example scripts are available in a separate repository [php-ai/php-ml-examples](
 - Issue Tracker: github.com/php-ai/php-ml/issues
 - Source Code: github.com/php-ai/php-ml
 
-After installation, you can launch the test suite in project root directory (you will need to install dev requirements with Composer)
-
-```
-bin/phpunit
-```
+You can find more about contributing in [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## License
 

docs/machine-learning/classification/k-nearest-neighbors.md

@@ -2,7 +2,7 @@
 
 Classifier implementing the k-nearest neighbors algorithm.
 
-### Constructor Parameters
+## Constructor Parameters
 
 * $k - number of nearest neighbors to scan (default: 3)
 * $distanceMetric - Distance object, default Euclidean (see [distance documentation](math/distance/))
@@ -12,7 +12,7 @@ $classifier = new KNearestNeighbors($k=4);
 $classifier = new KNearestNeighbors($k=3, new Minkowski($lambda=4));
 ```
 
-### Train
+## Train
 
 To train a classifier simply provide train samples and labels (as `array`). Example:
 
@@ -24,7 +24,7 @@ $classifier = new KNearestNeighbors();
 $classifier->train($samples, $labels);
 ```
 
-### Predict
+## Predict
 
 To predict sample label use `predict` method. You can provide one sample or array of samples:
 

docs/machine-learning/neural-network/backpropagation.md

@@ -0,0 +1,29 @@
+# Backpropagation
+
+Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent. 
+
+## Constructor Parameters
+
+* $network (Network) - network to train (for example MultilayerPerceptron instance)
+* $theta (int) - network theta parameter
+
+```
+use Phpml\NeuralNetwork\Network\MultilayerPerceptron;
+use Phpml\NeuralNetwork\Training\Backpropagation;
+
+$network = new MultilayerPerceptron([2, 2, 1]);
+$training = new Backpropagation($network);
+```
+
+## Training
+
+Example of XOR training:
+
+```
+$training->train(
+    $samples = [[1, 0], [0, 1], [1, 1], [0, 0]],
+    $targets = [[1], [1], [0], [0]],
+    $desiredError = 0.2,
+    $maxIteraions = 30000
+);
+```

docs/machine-learning/neural-network/multilayer-perceptron.md

@@ -0,0 +1,29 @@
+# MultilayerPerceptron
+
+A multilayer perceptron (MLP) is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs.
+
+## Constructor Parameters
+
+* $layers (array) - array with layers configuration, each value represent number of neurons in each layers
+* $activationFunction (ActivationFunction) - neuron activation function
+
+```
+use Phpml\NeuralNetwork\Network\MultilayerPerceptron;
+$mlp = new MultilayerPerceptron([2, 2, 1]);
+
+// 2 nodes in input layer, 2 nodes in first hidden layer and 1 node in output layer 
+```
+
+## Methods
+
+* setInput(array $input)
+* getOutput()
+* getLayers()
+* addLayer(Layer $layer)
+
+## Activation Functions
+
+* BinaryStep
+* Gaussian
+* HyperbolicTangent
+* Sigmoid (default)

mkdocs.yml

@@ -18,6 +18,9 @@ pages:
       - Classification Report: machine-learning/metric/classification-report.md
     - Workflow:
       - Pipeline: machine-learning/workflow/pipeline.md
+    - Neural Network:
+      - Multilayer Perceptron: machine-learning/neural-network/multilayer-perceptron.md
+      - Backpropagation training: machine-learning/neural-network/backpropagation.md
     - Cross Validation:
       - RandomSplit: machine-learning/cross-validation/random-split.md
       - Stratified Random Split: machine-learning/cross-validation/stratified-random-split.md

src/Phpml/Clustering/KMeans/Space.php

@@ -61,7 +61,7 @@ class Space extends SplObjectStorage
      */
     public function addPoint(array $coordinates, $data = null)
     {
-        return $this->attach($this->newPoint($coordinates), $data);
+        $this->attach($this->newPoint($coordinates), $data);
     }
 
     /**
@@ -74,7 +74,7 @@ class Space extends SplObjectStorage
             throw new InvalidArgumentException('can only attach points to spaces');
         }
 
-        return parent::attach($point, $data);
+        parent::attach($point, $data);
     }
 
     /**
@@ -230,8 +230,8 @@ class Space extends SplObjectStorage
     protected function initializeKMPPClusters(int $clustersNumber)
     {
         $clusters = [];
-        $position = rand(1, count($this));
-        for ($i = 1, $this->rewind(); $i < $position && $this->valid(); $i++, $this->next());
+        $this->rewind();
+
         $clusters[] = new Cluster($this, $this->current()->getCoordinates());
 
         $distances = new SplObjectStorage();

src/Phpml/Exception/InvalidArgumentException.php

@@ -73,4 +73,20 @@ class InvalidArgumentException extends \Exception
     {
         return new self(sprintf('Can\'t find %s language for StopWords', $language));
     }
+
+    /**
+     * @return InvalidArgumentException
+     */
+    public static function invalidLayerNodeClass()
+    {
+        return new self('Layer node class must implement Node interface');
+    }
+
+    /**
+     * @return InvalidArgumentException
+     */
+    public static function invalidLayersNumber()
+    {
+        return new self('Provide at least 2 layers: 1 input and 1 output');
+    }
 }

src/Phpml/Math/Matrix.php

@@ -193,7 +193,8 @@ class Matrix
         $product = [];
         $multiplier = $matrix->toArray();
         for ($i = 0; $i < $this->rows; ++$i) {
-            for ($j = 0; $j < $matrix->getColumns(); ++$j) {
+            $columns = $matrix->getColumns();
+            for ($j = 0; $j < $columns; ++$j) {
                 $product[$i][$j] = 0;
                 for ($k = 0; $k < $this->columns; ++$k) {
                     $product[$i][$j] += $this->matrix[$i][$k] * $multiplier[$k][$j];

src/Phpml/Math/Product.php

@@ -16,7 +16,9 @@ class Product
     {
         $product = 0;
         foreach ($a as $index => $value) {
-            $product += $value * $b[$index];
+            if (is_numeric($value) && is_numeric($b[$index])) {
+                $product += $value * $b[$index];
+            }
         }
 
         return $product;

src/Phpml/NeuralNetwork/ActivationFunction.php

@@ -0,0 +1,15 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork;
+
+interface ActivationFunction
+{
+    /**
+     * @param float|int $value
+     *
+     * @return float
+     */
+    public function compute($value): float;
+}

src/Phpml/NeuralNetwork/ActivationFunction/BinaryStep.php

@@ -0,0 +1,20 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction;
+
+class BinaryStep implements ActivationFunction
+{
+    /**
+     * @param float|int $value
+     *
+     * @return float
+     */
+    public function compute($value): float
+    {
+        return $value >= 0 ? 1.0 : 0.0;
+    }
+}

src/Phpml/NeuralNetwork/ActivationFunction/Gaussian.php

@@ -0,0 +1,20 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction;
+
+class Gaussian implements ActivationFunction
+{
+    /**
+     * @param float|int $value
+     *
+     * @return float
+     */
+    public function compute($value): float
+    {
+        return exp(-pow($value, 2));
+    }
+}

src/Phpml/NeuralNetwork/ActivationFunction/HyperbolicTangent.php

@@ -0,0 +1,33 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction;
+
+class HyperbolicTangent implements ActivationFunction
+{
+    /**
+     * @var float
+     */
+    private $beta;
+
+    /**
+     * @param float $beta
+     */
+    public function __construct($beta = 1.0)
+    {
+        $this->beta = $beta;
+    }
+
+    /**
+     * @param float|int $value
+     *
+     * @return float
+     */
+    public function compute($value): float
+    {
+        return tanh($this->beta * $value);
+    }
+}

src/Phpml/NeuralNetwork/ActivationFunction/Sigmoid.php

@@ -0,0 +1,33 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction;
+
+class Sigmoid implements ActivationFunction
+{
+    /**
+     * @var float
+     */
+    private $beta;
+
+    /**
+     * @param float $beta
+     */
+    public function __construct($beta = 1.0)
+    {
+        $this->beta = $beta;
+    }
+
+    /**
+     * @param float|int $value
+     *
+     * @return float
+     */
+    public function compute($value): float
+    {
+        return 1 / (1 + exp(-$this->beta * $value));
+    }
+}

src/Phpml/NeuralNetwork/Layer.php

@@ -0,0 +1,65 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork;
+
+use Phpml\Exception\InvalidArgumentException;
+use Phpml\NeuralNetwork\Node\Neuron;
+
+class Layer
+{
+    /**
+     * @var Node[]
+     */
+    private $nodes = [];
+
+    /**
+     * @param int                     $nodesNumber
+     * @param string                  $nodeClass
+     * @param ActivationFunction|null $activationFunction
+     *
+     * @throws InvalidArgumentException
+     */
+    public function __construct(int $nodesNumber = 0, string $nodeClass = Neuron::class, ActivationFunction $activationFunction = null)
+    {
+        if (!in_array(Node::class, class_implements($nodeClass))) {
+            throw InvalidArgumentException::invalidLayerNodeClass();
+        }
+
+        for ($i = 0; $i < $nodesNumber; ++$i) {
+            $this->nodes[] = $this->createNode($nodeClass, $activationFunction);
+        }
+    }
+
+    /**
+     * @param string                  $nodeClass
+     * @param ActivationFunction|null $activationFunction
+     *
+     * @return Neuron
+     */
+    private function createNode(string $nodeClass, ActivationFunction $activationFunction = null)
+    {
+        if (Neuron::class == $nodeClass) {
+            return new Neuron($activationFunction);
+        }
+
+        return new $nodeClass();
+    }
+
+    /**
+     * @param Node $node
+     */
+    public function addNode(Node $node)
+    {
+        $this->nodes[] = $node;
+    }
+
+    /**
+     * @return Node[]
+     */
+    public function getNodes()
+    {
+        return $this->nodes;
+    }
+}

src/Phpml/NeuralNetwork/Network.php

@@ -0,0 +1,30 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork;
+
+interface Network
+{
+    /**
+     * @param mixed $input
+     *
+     * @return self
+     */
+    public function setInput($input);
+
+    /**
+     * @return array
+     */
+    public function getOutput(): array;
+
+    /**
+     * @param Layer $layer
+     */
+    public function addLayer(Layer $layer);
+
+    /**
+     * @return Layer[]
+     */
+    public function getLayers(): array;
+}

src/Phpml/NeuralNetwork/Network/LayeredNetwork.php

@@ -0,0 +1,81 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Network;
+
+use Phpml\NeuralNetwork\Layer;
+use Phpml\NeuralNetwork\Network;
+use Phpml\NeuralNetwork\Node\Input;
+use Phpml\NeuralNetwork\Node\Neuron;
+
+abstract class LayeredNetwork implements Network
+{
+    /**
+     * @var Layer[]
+     */
+    protected $layers;
+
+    /**
+     * @param Layer $layer
+     */
+    public function addLayer(Layer $layer)
+    {
+        $this->layers[] = $layer;
+    }
+
+    /**
+     * @return Layer[]
+     */
+    public function getLayers(): array
+    {
+        return $this->layers;
+    }
+
+    /**
+     * @return Layer
+     */
+    public function getOutputLayer(): Layer
+    {
+        return $this->layers[count($this->layers) - 1];
+    }
+
+    /**
+     * @return array
+     */
+    public function getOutput(): array
+    {
+        $result = [];
+        foreach ($this->getOutputLayer()->getNodes() as $neuron) {
+            $result[] = $neuron->getOutput();
+        }
+
+        return $result;
+    }
+
+    /**
+     * @param mixed $input
+     *
+     * @return $this
+     */
+    public function setInput($input)
+    {
+        $firstLayer = $this->layers[0];
+
+        foreach ($firstLayer->getNodes() as $key => $neuron) {
+            if ($neuron instanceof Input) {
+                $neuron->setInput($input[$key]);
+            }
+        }
+
+        foreach ($this->getLayers() as $layer) {
+            foreach ($layer->getNodes() as $node) {
+                if ($node instanceof Neuron) {
+                    $node->refresh();
+                }
+            }
+        }
+
+        return $this;
+    }
+}

src/Phpml/NeuralNetwork/Network/MultilayerPerceptron.php

@@ -0,0 +1,95 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Network;
+
+use Phpml\Exception\InvalidArgumentException;
+use Phpml\NeuralNetwork\ActivationFunction;
+use Phpml\NeuralNetwork\Layer;
+use Phpml\NeuralNetwork\Node\Bias;
+use Phpml\NeuralNetwork\Node\Input;
+use Phpml\NeuralNetwork\Node\Neuron;
+use Phpml\NeuralNetwork\Node\Neuron\Synapse;
+
+class MultilayerPerceptron extends LayeredNetwork
+{
+    /**
+     * @param array                   $layers
+     * @param ActivationFunction|null $activationFunction
+     *
+     * @throws InvalidArgumentException
+     */
+    public function __construct(array $layers, ActivationFunction $activationFunction = null)
+    {
+        if (count($layers) < 2) {
+            throw InvalidArgumentException::invalidLayersNumber();
+        }
+
+        $this->addInputLayer(array_shift($layers));
+        $this->addNeuronLayers($layers, $activationFunction);
+        $this->addBiasNodes();
+        $this->generateSynapses();
+    }
+
+    /**
+     * @param int $nodes
+     */
+    private function addInputLayer(int $nodes)
+    {
+        $this->addLayer(new Layer($nodes, Input::class));
+    }
+
+    /**
+     * @param array                   $layers
+     * @param ActivationFunction|null $activationFunction
+     */
+    private function addNeuronLayers(array $layers, ActivationFunction $activationFunction = null)
+    {
+        foreach ($layers as $neurons) {
+            $this->addLayer(new Layer($neurons, Neuron::class, $activationFunction));
+        }
+    }
+
+    private function generateSynapses()
+    {
+        $layersNumber = count($this->layers) - 1;
+        for ($i = 0; $i < $layersNumber; ++$i) {
+            $currentLayer = $this->layers[$i];
+            $nextLayer = $this->layers[$i + 1];
+            $this->generateLayerSynapses($nextLayer, $currentLayer);
+        }
+    }
+
+    private function addBiasNodes()
+    {
+        $biasLayers = count($this->layers) - 1;
+        for ($i = 0;$i < $biasLayers;++$i) {
+            $this->layers[$i]->addNode(new Bias());
+        }
+    }
+
+    /**
+     * @param Layer $nextLayer
+     * @param Layer $currentLayer
+     */
+    private function generateLayerSynapses(Layer $nextLayer, Layer $currentLayer)
+    {
+        foreach ($nextLayer->getNodes() as $nextNeuron) {
+            if ($nextNeuron instanceof Neuron) {
+                $this->generateNeuronSynapses($currentLayer, $nextNeuron);
+            }
+        }
+    }
+
+    /**
+     * @param Layer  $currentLayer
+     * @param Neuron $nextNeuron
+     */
+    private function generateNeuronSynapses(Layer $currentLayer, Neuron $nextNeuron)
+    {
+        foreach ($currentLayer->getNodes() as $currentNeuron) {
+            $nextNeuron->addSynapse(new Synapse($currentNeuron));
+        }
+    }
+}

src/Phpml/NeuralNetwork/Node.php

@@ -0,0 +1,13 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork;
+
+interface Node
+{
+    /**
+     * @return float
+     */
+    public function getOutput(): float;
+}

src/Phpml/NeuralNetwork/Node/Bias.php

@@ -0,0 +1,18 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\Node;
+
+class Bias implements Node
+{
+    /**
+     * @return float
+     */
+    public function getOutput(): float
+    {
+        return 1.0;
+    }
+}

src/Phpml/NeuralNetwork/Node/Input.php

@@ -0,0 +1,39 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\Node;
+
+class Input implements Node
+{
+    /**
+     * @var float
+     */
+    private $input;
+
+    /**
+     * @param float $input
+     */
+    public function __construct(float $input = 0.0)
+    {
+        $this->input = $input;
+    }
+
+    /**
+     * @return float
+     */
+    public function getOutput(): float
+    {
+        return $this->input;
+    }
+
+    /**
+     * @param float $input
+     */
+    public function setInput(float $input)
+    {
+        $this->input = $input;
+    }
+}

src/Phpml/NeuralNetwork/Node/Neuron.php

@@ -0,0 +1,75 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\ActivationFunction;
+use Phpml\NeuralNetwork\Node\Neuron\Synapse;
+use Phpml\NeuralNetwork\Node;
+
+class Neuron implements Node
+{
+    /**
+     * @var Synapse[]
+     */
+    protected $synapses;
+
+    /**
+     * @var ActivationFunction
+     */
+    protected $activationFunction;
+
+    /**
+     * @var float
+     */
+    protected $output;
+
+    /**
+     * @param ActivationFunction|null $activationFunction
+     */
+    public function __construct(ActivationFunction $activationFunction = null)
+    {
+        $this->activationFunction = $activationFunction ?: new ActivationFunction\Sigmoid();
+        $this->synapses = [];
+        $this->output = 0;
+    }
+
+    /**
+     * @param Synapse $synapse
+     */
+    public function addSynapse(Synapse $synapse)
+    {
+        $this->synapses[] = $synapse;
+    }
+
+    /**
+     * @return Synapse[]
+     */
+    public function getSynapses()
+    {
+        return $this->synapses;
+    }
+
+    /**
+     * @return float
+     */
+    public function getOutput(): float
+    {
+        if (0 === $this->output) {
+            $sum = 0;
+            foreach ($this->synapses as $synapse) {
+                $sum += $synapse->getOutput();
+            }
+
+            $this->output = $this->activationFunction->compute($sum);
+        }
+
+        return $this->output;
+    }
+
+    public function refresh()
+    {
+        $this->output = 0;
+    }
+}

src/Phpml/NeuralNetwork/Node/Neuron/Synapse.php

@@ -0,0 +1,70 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Node\Neuron;
+
+use Phpml\NeuralNetwork\Node;
+
+class Synapse
+{
+    /**
+     * @var float
+     */
+    protected $weight;
+
+    /**
+     * @var Node
+     */
+    protected $node;
+
+    /**
+     * @param Node       $node
+     * @param float|null $weight
+     */
+    public function __construct(Node $node, float $weight = null)
+    {
+        $this->node = $node;
+        $this->weight = $weight ?: $this->generateRandomWeight();
+    }
+
+    /**
+     * @return float
+     */
+    protected function generateRandomWeight(): float
+    {
+        return 1 / rand(5, 25) * (rand(0, 1) ? -1 : 1);
+    }
+
+    /**
+     * @return float
+     */
+    public function getOutput(): float
+    {
+        return $this->weight * $this->node->getOutput();
+    }
+
+    /**
+     * @param float $delta
+     */
+    public function changeWeight($delta)
+    {
+        $this->weight += $delta;
+    }
+
+    /**
+     * @return float
+     */
+    public function getWeight()
+    {
+        return $this->weight;
+    }
+
+    /**
+     * @return Node
+     */
+    public function getNode()
+    {
+        return $this->node;
+    }
+}

src/Phpml/NeuralNetwork/Training.php

@@ -0,0 +1,16 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork;
+
+interface Training
+{
+    /**
+     * @param array $samples
+     * @param array $targets
+     * @param float $desiredError
+     * @param int   $maxIterations
+     */
+    public function train(array $samples, array $targets, float $desiredError = 0.001, int $maxIterations = 10000);
+}

src/Phpml/NeuralNetwork/Training/Backpropagation.php

@@ -0,0 +1,160 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Training;
+
+use Phpml\NeuralNetwork\Network;
+use Phpml\NeuralNetwork\Node\Neuron;
+use Phpml\NeuralNetwork\Training;
+use Phpml\NeuralNetwork\Training\Backpropagation\Sigma;
+
+class Backpropagation implements Training
+{
+    /**
+     * @var Network
+     */
+    private $network;
+
+    /**
+     * @var int
+     */
+    private $theta;
+
+    /**
+     * @var array
+     */
+    private $sigmas;
+
+    /**
+     * @param Network $network
+     * @param int     $theta
+     */
+    public function __construct(Network $network, int $theta = 1)
+    {
+        $this->network = $network;
+        $this->theta = $theta;
+    }
+
+    /**
+     * @param array $samples
+     * @param array $targets
+     * @param float $desiredError
+     * @param int   $maxIterations
+     */
+    public function train(array $samples, array $targets, float $desiredError = 0.001, int $maxIterations = 10000)
+    {
+        for ($i = 0; $i < $maxIterations; ++$i) {
+            $resultsWithinError = $this->trainSamples($samples, $targets, $desiredError);
+
+            if ($resultsWithinError == count($samples)) {
+                break;
+            }
+        }
+    }
+
+    /**
+     * @param array $samples
+     * @param array $targets
+     * @param float $desiredError
+     *
+     * @return int
+     */
+    private function trainSamples(array $samples, array $targets, float $desiredError): int
+    {
+        $resultsWithinError = 0;
+        foreach ($targets as $key => $target) {
+            $result = $this->network->setInput($samples[$key])->getOutput();
+
+            if ($this->isResultWithinError($result, $target, $desiredError)) {
+                ++$resultsWithinError;
+            } else {
+                $this->trainSample($samples[$key], $target);
+            }
+        }
+
+        return $resultsWithinError;
+    }
+
+    /**
+     * @param array $sample
+     * @param array $target
+     */
+    private function trainSample(array $sample, array $target)
+    {
+        $this->network->setInput($sample)->getOutput();
+        $this->sigmas = [];
+
+        $layers = $this->network->getLayers();
+        $layersNumber = count($layers);
+
+        for ($i = $layersNumber; $i > 1; --$i) {
+            foreach ($layers[$i - 1]->getNodes() as $key => $neuron) {
+                if ($neuron instanceof Neuron) {
+                    $sigma = $this->getSigma($neuron, $target, $key, $i == $layersNumber);
+                    foreach ($neuron->getSynapses() as $synapse) {
+                        $synapse->changeWeight($this->theta * $sigma * $synapse->getNode()->getOutput());
+                    }
+                }
+            }
+        }
+    }
+
+    /**
+     * @param Neuron $neuron
+     * @param array  $target
+     * @param int    $key
+     * @param bool   $lastLayer
+     *
+     * @return float
+     */
+    private function getSigma(Neuron $neuron, array $target, int $key, bool $lastLayer): float
+    {
+        $neuronOutput = $neuron->getOutput();
+        $sigma = $neuronOutput * (1 - $neuronOutput);
+
+        if ($lastLayer) {
+            $sigma *= ($target[$key] - $neuronOutput);
+        } else {
+            $sigma *= $this->getPrevSigma($neuron);
+        }
+
+        $this->sigmas[] = new Sigma($neuron, $sigma);
+
+        return $sigma;
+    }
+
+    /**
+     * @param Neuron $neuron
+     * 
+     * @return float
+     */
+    private function getPrevSigma(Neuron $neuron): float
+    {
+        $sigma = 0.0;
+
+        foreach ($this->sigmas as $neuronSigma) {
+            $sigma += $neuronSigma->getSigmaForNeuron($neuron);
+        }
+
+        return $sigma;
+    }
+
+    /**
+     * @param array $result
+     * @param array $target
+     * @param float $desiredError
+     *
+     * @return bool
+     */
+    private function isResultWithinError(array $result, array $target, float $desiredError)
+    {
+        foreach ($target as $key => $value) {
+            if ($result[$key] > $value + $desiredError || $result[$key] < $value - $desiredError) {
+                return false;
+            }
+        }
+
+        return true;
+    }
+}

src/Phpml/NeuralNetwork/Training/Backpropagation/Sigma.php

@@ -0,0 +1,64 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\NeuralNetwork\Training\Backpropagation;
+
+use Phpml\NeuralNetwork\Node\Neuron;
+
+class Sigma
+{
+    /**
+     * @var Neuron
+     */
+    private $neuron;
+
+    /**
+     * @var float
+     */
+    private $sigma;
+
+    /**
+     * @param Neuron $neuron
+     * @param float  $sigma
+     */
+    public function __construct(Neuron $neuron, $sigma)
+    {
+        $this->neuron = $neuron;
+        $this->sigma = $sigma;
+    }
+
+    /**
+     * @return Neuron
+     */
+    public function getNeuron()
+    {
+        return $this->neuron;
+    }
+
+    /**
+     * @return float
+     */
+    public function getSigma()
+    {
+        return $this->sigma;
+    }
+
+    /**
+     * @param Neuron $neuron
+     *
+     * @return float
+     */
+    public function getSigmaForNeuron(Neuron $neuron): float
+    {
+        $sigma = 0.0;
+
+        foreach ($this->neuron->getSynapses() as $synapse) {
+            if ($synapse->getNode() == $neuron) {
+                $sigma += $synapse->getWeight() * $this->getSigma();
+            }
+        }
+
+        return $sigma;
+    }
+}

src/Phpml/Regression/MLPRegressor.php

@@ -0,0 +1,80 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace Phpml\Regression;
+
+use Phpml\Helper\Predictable;
+use Phpml\NeuralNetwork\ActivationFunction;
+use Phpml\NeuralNetwork\Network\MultilayerPerceptron;
+use Phpml\NeuralNetwork\Training\Backpropagation;
+
+class MLPRegressor implements Regression
+{
+    use Predictable;
+
+    /**
+     * @var MultilayerPerceptron
+     */
+    private $perceptron;
+
+    /**
+     * @var array
+     */
+    private $hiddenLayers;
+
+    /**
+     * @var float
+     */
+    private $desiredError;
+
+    /**
+     * @var int
+     */
+    private $maxIterations;
+
+    /**
+     * @var ActivationFunction
+     */
+    private $activationFunction;
+
+    /**
+     * @param array              $hiddenLayers
+     * @param float              $desiredError
+     * @param int                $maxIterations
+     * @param ActivationFunction $activationFunction
+     */
+    public function __construct(array $hiddenLayers = [10], float $desiredError = 0.01, int $maxIterations = 10000, ActivationFunction $activationFunction = null)
+    {
+        $this->hiddenLayers = $hiddenLayers;
+        $this->desiredError = $desiredError;
+        $this->maxIterations = $maxIterations;
+        $this->activationFunction = $activationFunction;
+    }
+
+    /**
+     * @param array $samples
+     * @param array $targets
+     */
+    public function train(array $samples, array $targets)
+    {
+        $layers = $this->hiddenLayers;
+        array_unshift($layers, count($samples[0]));
+        $layers[] = count($targets[0]);
+
+        $this->perceptron = new MultilayerPerceptron($layers, $this->activationFunction);
+
+        $trainer = new Backpropagation($this->perceptron);
+        $trainer->train($samples, $targets, $this->desiredError, $this->maxIterations);
+    }
+
+    /**
+     * @param array $sample
+     *
+     * @return array
+     */
+    protected function predictSample(array $sample)
+    {
+        return $this->perceptron->setInput($sample)->getOutput();
+    }
+}

tests/Phpml/Math/ProductTest.php

@@ -13,5 +13,8 @@ class ProductTest extends \PHPUnit_Framework_TestCase
         $this->assertEquals(10, Product::scalar([2, 3], [-1, 4]));
         $this->assertEquals(-0.1, Product::scalar([1, 4, 1], [-2, 0.5, -0.1]));
         $this->assertEquals(8, Product::scalar([2], [4]));
+
+        //test for non numeric values
+        $this->assertEquals(0, Product::scalar(['', null, [], new \stdClass()], [null]));
     }
 }

tests/Phpml/NeuralNetwork/ActivationFunction/BinaryStepTest.php

@@ -0,0 +1,35 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction\BinaryStep;
+
+class BinaryStepTest extends \PHPUnit_Framework_TestCase
+{
+    /**
+     * @param $expected
+     * @param $value
+     *
+     * @dataProvider binaryStepProvider
+     */
+    public function testBinaryStepActivationFunction($expected, $value)
+    {
+        $binaryStep = new BinaryStep();
+
+        $this->assertEquals($expected, $binaryStep->compute($value));
+    }
+
+    /**
+     * @return array
+     */
+    public function binaryStepProvider()
+    {
+        return [
+            [1, 1],
+            [1, 0],
+            [0, -0.1],
+        ];
+    }
+}

tests/Phpml/NeuralNetwork/ActivationFunction/GaussianTest.php

@@ -0,0 +1,37 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction\Gaussian;
+
+class GaussianTest extends \PHPUnit_Framework_TestCase
+{
+    /**
+     * @param $expected
+     * @param $value
+     *
+     * @dataProvider gaussianProvider
+     */
+    public function testGaussianActivationFunction($expected, $value)
+    {
+        $gaussian = new Gaussian();
+
+        $this->assertEquals($expected, $gaussian->compute($value), '', 0.001);
+    }
+
+    /**
+     * @return array
+     */
+    public function gaussianProvider()
+    {
+        return [
+            [0.367, 1],
+            [1, 0],
+            [0.367, -1],
+            [0, 3],
+            [0, -3],
+        ];
+    }
+}

tests/Phpml/NeuralNetwork/ActivationFunction/HyperboliTangentTest.php

@@ -0,0 +1,39 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction\HyperbolicTangent;
+
+class HyperboliTangentTest extends \PHPUnit_Framework_TestCase
+{
+    /**
+     * @param $beta
+     * @param $expected
+     * @param $value
+     *
+     * @dataProvider tanhProvider
+     */
+    public function testHyperbolicTangentActivationFunction($beta, $expected, $value)
+    {
+        $tanh = new HyperbolicTangent($beta);
+
+        $this->assertEquals($expected, $tanh->compute($value), '', 0.001);
+    }
+
+    /**
+     * @return array
+     */
+    public function tanhProvider()
+    {
+        return [
+            [1.0, 0.761, 1],
+            [1.0, 0, 0],
+            [1.0, 1, 4],
+            [1.0, -1, -4],
+            [0.5, 0.462, 1],
+            [0.3, 0, 0],
+        ];
+    }
+}

tests/Phpml/NeuralNetwork/ActivationFunction/SigmoidTest.php

@@ -0,0 +1,39 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\ActivationFunction;
+
+use Phpml\NeuralNetwork\ActivationFunction\Sigmoid;
+
+class SigmoidTest extends \PHPUnit_Framework_TestCase
+{
+    /**
+     * @param $beta
+     * @param $expected
+     * @param $value
+     *
+     * @dataProvider sigmoidProvider
+     */
+    public function testSigmoidActivationFunction($beta, $expected, $value)
+    {
+        $sigmoid = new Sigmoid($beta);
+
+        $this->assertEquals($expected, $sigmoid->compute($value), '', 0.001);
+    }
+
+    /**
+     * @return array
+     */
+    public function sigmoidProvider()
+    {
+        return [
+            [1.0, 1, 7.25],
+            [2.0, 1, 3.75],
+            [1.0, 0.5, 0],
+            [0.5, 0.5, 0],
+            [1.0, 0, -7.25],
+            [2.0, 0, -3.75],
+        ];
+    }
+}

tests/Phpml/NeuralNetwork/LayerTest.php

@@ -0,0 +1,56 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork;
+
+use Phpml\NeuralNetwork\Node\Bias;
+use Phpml\NeuralNetwork\Layer;
+use Phpml\NeuralNetwork\Node\Neuron;
+
+class LayerTest extends \PHPUnit_Framework_TestCase
+{
+    public function testLayerInitialization()
+    {
+        $layer = new Layer();
+
+        $this->assertEquals([], $layer->getNodes());
+    }
+
+    public function testLayerInitializationWithDefaultNodesType()
+    {
+        $layer = new Layer($number = 5);
+
+        $this->assertCount($number, $layer->getNodes());
+        foreach ($layer->getNodes() as $node) {
+            $this->assertInstanceOf(Neuron::class, $node);
+        }
+    }
+
+    public function testLayerInitializationWithExplicitNodesType()
+    {
+        $layer = new Layer($number = 5, $class = Bias::class);
+
+        $this->assertCount($number, $layer->getNodes());
+        foreach ($layer->getNodes() as $node) {
+            $this->assertInstanceOf($class, $node);
+        }
+    }
+
+    /**
+     * @expectedException \Phpml\Exception\InvalidArgumentException
+     */
+    public function testThrowExceptionOnInvalidNodeClass()
+    {
+        new Layer(1, \stdClass::class);
+    }
+
+    public function testAddNodesToLayer()
+    {
+        $layer = new Layer();
+        $layer->addNode($node1 = new Neuron());
+        $layer->addNode($node2 = new Neuron());
+
+        $this->assertEquals([$node1, $node2], $layer->getNodes());
+    }
+}

tests/Phpml/NeuralNetwork/Network/LayeredNetworkTest.php

@@ -0,0 +1,53 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Network;
+
+use Phpml\NeuralNetwork\Layer;
+use Phpml\NeuralNetwork\Network\LayeredNetwork;
+use Phpml\NeuralNetwork\Node\Input;
+
+class LayeredNetworkTest extends \PHPUnit_Framework_TestCase
+{
+    public function testLayersSettersAndGetters()
+    {
+        $network = $this->getLayeredNetworkMock();
+
+        $network->addLayer($layer1 = new Layer());
+        $network->addLayer($layer2 = new Layer());
+
+        $this->assertEquals([$layer1, $layer2], $network->getLayers());
+    }
+
+    public function testGetLastLayerAsOutputLayer()
+    {
+        $network = $this->getLayeredNetworkMock();
+        $network->addLayer($layer1 = new Layer());
+
+        $this->assertEquals($layer1, $network->getOutputLayer());
+
+        $network->addLayer($layer2 = new Layer());
+        $this->assertEquals($layer2, $network->getOutputLayer());
+    }
+
+    public function testSetInputAndGetOutput()
+    {
+        $network = $this->getLayeredNetworkMock();
+        $network->addLayer(new Layer(2, Input::class));
+
+        $network->setInput($input = [34, 43]);
+        $this->assertEquals($input, $network->getOutput());
+
+        $network->addLayer(new Layer(1));
+        $this->assertEquals([0.5], $network->getOutput());
+    }
+
+    /**
+     * @return LayeredNetwork
+     */
+    private function getLayeredNetworkMock()
+    {
+        return $this->getMockForAbstractClass(LayeredNetwork::class);
+    }
+}

tests/Phpml/NeuralNetwork/Network/MultilayerPerceptronTest.php

@@ -0,0 +1,73 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Network;
+
+use Phpml\NeuralNetwork\Network\MultilayerPerceptron;
+use Phpml\NeuralNetwork\Node\Neuron;
+
+class MultilayerPerceptronTest extends \PHPUnit_Framework_TestCase
+{
+    public function testMultilayerPerceptronLayersInitialization()
+    {
+        $mlp = new MultilayerPerceptron([2, 2, 1]);
+
+        $this->assertCount(3, $mlp->getLayers());
+
+        $layers = $mlp->getLayers();
+
+        // input layer
+        $this->assertCount(3, $layers[0]->getNodes());
+        $this->assertNotContainsOnly(Neuron::class, $layers[0]->getNodes());
+
+        // hidden layer
+        $this->assertCount(3, $layers[1]->getNodes());
+        $this->assertNotContainsOnly(Neuron::class, $layers[0]->getNodes());
+
+        // output layer
+        $this->assertCount(1, $layers[2]->getNodes());
+        $this->assertContainsOnly(Neuron::class, $layers[2]->getNodes());
+    }
+
+    public function testSynapsesGeneration()
+    {
+        $mlp = new MultilayerPerceptron([2, 2, 1]);
+        $layers = $mlp->getLayers();
+
+        foreach ($layers[1]->getNodes() as $node) {
+            if ($node instanceof Neuron) {
+                $synapses = $node->getSynapses();
+                $this->assertCount(3, $synapses);
+
+                $synapsesNodes = $this->getSynapsesNodes($synapses);
+                foreach ($layers[0]->getNodes() as $prevNode) {
+                    $this->assertContains($prevNode, $synapsesNodes);
+                }
+            }
+        }
+    }
+
+    /**
+     * @param array $synapses
+     *
+     * @return array
+     */
+    private function getSynapsesNodes(array $synapses): array
+    {
+        $nodes = [];
+        foreach ($synapses as $synapse) {
+            $nodes[] = $synapse->getNode();
+        }
+
+        return $nodes;
+    }
+
+    /**
+     * @expectedException \Phpml\Exception\InvalidArgumentException
+     */
+    public function testThrowExceptionOnInvalidLayersNumber()
+    {
+        new MultilayerPerceptron([2]);
+    }
+}

tests/Phpml/NeuralNetwork/Node/BiasTest.php

@@ -0,0 +1,17 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\Node\Bias;
+
+class BiasTest extends \PHPUnit_Framework_TestCase
+{
+    public function testBiasOutput()
+    {
+        $bias = new Bias();
+
+        $this->assertEquals(1.0, $bias->getOutput());
+    }
+}

tests/Phpml/NeuralNetwork/Node/InputTest.php

@@ -0,0 +1,27 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\Node\Input;
+
+class InputTest extends \PHPUnit_Framework_TestCase
+{
+    public function testInputInitialization()
+    {
+        $input = new Input();
+        $this->assertEquals(0.0, $input->getOutput());
+
+        $input = new Input($value = 9.6);
+        $this->assertEquals($value, $input->getOutput());
+    }
+
+    public function testSetInput()
+    {
+        $input = new Input();
+        $input->setInput($value = 6.9);
+
+        $this->assertEquals($value, $input->getOutput());
+    }
+}

tests/Phpml/NeuralNetwork/Node/Neuron/SynapseTest.php

@@ -0,0 +1,52 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Node\Neuron;
+
+use Phpml\NeuralNetwork\Node\Neuron\Synapse;
+use Phpml\NeuralNetwork\Node\Neuron;
+
+class SynapseTest extends \PHPUnit_Framework_TestCase
+{
+    public function testSynapseInitialization()
+    {
+        $node = $this->getNodeMock($nodeOutput = 0.5);
+
+        $synapse = new Synapse($node, $weight = 0.75);
+
+        $this->assertEquals($node, $synapse->getNode());
+        $this->assertEquals($weight, $synapse->getWeight());
+        $this->assertEquals($weight * $nodeOutput, $synapse->getOutput());
+
+        $synapse = new Synapse($node);
+
+        $this->assertInternalType('float', $synapse->getWeight());
+    }
+
+    public function testSynapseWeightChange()
+    {
+        $node = $this->getNodeMock();
+        $synapse = new Synapse($node, $weight = 0.75);
+        $synapse->changeWeight(1.0);
+
+        $this->assertEquals(1.75, $synapse->getWeight());
+
+        $synapse->changeWeight(-2.0);
+
+        $this->assertEquals(-0.25, $synapse->getWeight());
+    }
+
+    /**
+     * @param int $output
+     *
+     * @return \PHPUnit_Framework_MockObject_MockObject
+     */
+    private function getNodeMock($output = 1)
+    {
+        $node = $this->getMock(Neuron::class);
+        $node->method('getOutput')->willReturn($nodeOutput = 0.5);
+
+        return $node;
+    }
+}

tests/Phpml/NeuralNetwork/Node/NeuronTest.php

@@ -0,0 +1,65 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Node;
+
+use Phpml\NeuralNetwork\ActivationFunction\BinaryStep;
+use Phpml\NeuralNetwork\Node\Neuron;
+use Phpml\NeuralNetwork\Node\Neuron\Synapse;
+
+class NeuronTest extends \PHPUnit_Framework_TestCase
+{
+    public function testNeuronInitialization()
+    {
+        $neuron = new Neuron();
+
+        $this->assertEquals([], $neuron->getSynapses());
+        $this->assertEquals(0.5, $neuron->getOutput());
+    }
+
+    public function testNeuronActivationFunction()
+    {
+        $activationFunction = $this->getMock(BinaryStep::class);
+        $activationFunction->method('compute')->with(0)->willReturn($output = 0.69);
+
+        $neuron = new Neuron($activationFunction);
+
+        $this->assertEquals($output, $neuron->getOutput());
+    }
+
+    public function testNeuronWithSynapse()
+    {
+        $neuron = new Neuron();
+        $neuron->addSynapse($synapse = $this->getSynapseMock());
+
+        $this->assertEquals([$synapse], $neuron->getSynapses());
+        $this->assertEquals(0.88, $neuron->getOutput(), '', 0.01);
+    }
+
+    public function testNeuronRefresh()
+    {
+        $neuron = new Neuron();
+        $neuron->getOutput();
+        $neuron->addSynapse($this->getSynapseMock());
+
+        $this->assertEquals(0.5, $neuron->getOutput(), '', 0.01);
+
+        $neuron->refresh();
+
+        $this->assertEquals(0.88, $neuron->getOutput(), '', 0.01);
+    }
+
+    /**
+     * @param int $output
+     *
+     * @return Synapse|\PHPUnit_Framework_MockObject_MockObject
+     */
+    private function getSynapseMock($output = 2)
+    {
+        $synapse = $this->getMock(Synapse::class, [], [], '', false);
+        $synapse->method('getOutput')->willReturn($output);
+
+        return $synapse;
+    }
+}

tests/Phpml/NeuralNetwork/Training/BackpropagationTest.php

@@ -0,0 +1,29 @@
+<?php
+
+declare (strict_types = 1);
+
+namespace tests\Phpml\NeuralNetwork\Training;
+
+use Phpml\NeuralNetwork\Network\MultilayerPerceptron;
+use Phpml\NeuralNetwork\Training\Backpropagation;
+
+class BackpropagationTest extends \PHPUnit_Framework_TestCase
+{
+    public function testBackpropagationForXORLearning()
+    {
+        $network = new MultilayerPerceptron([2, 2, 1]);
+        $training = new Backpropagation($network);
+
+        $training->train(
+            [[1, 0], [0, 1], [1, 1], [0, 0]],
+            [[1], [1], [0], [0]],
+            $desiredError = 0.2,
+            30000
+        );
+
+        $this->assertEquals(0, $network->setInput([1, 1])->getOutput()[0], '', $desiredError);
+        $this->assertEquals(0, $network->setInput([0, 0])->getOutput()[0], '', $desiredError);
+        $this->assertEquals(1, $network->setInput([1, 0])->getOutput()[0], '', $desiredError);
+        $this->assertEquals(1, $network->setInput([0, 1])->getOutput()[0], '', $desiredError);
+    }
+}