TensorFlow using Python

Some background….

TensorFlow by Google is one of the hottest libraries for machine learning out there. It uses a graph for mathematical computation where the nodes are mathematical operations and the edges are the data arrays which are multidimensional called Tensors. It is lazy and only compute the graph when it must,so you will need to tell it to run ,We user the interface that is written in python but the code actually run’s the graph in a super high performance c++ program so it is much more faster then any python library.  Another advantage is thae ability to deploy it on a single/multi machine CPU, GPU,clusters it can even be deployed on a mobile phone. Tensorflow main competitors are Theano, Caffe and Torch.

How to install TensorFlow

Before we start playing we will need to install the Tensorflow package for CPU

pip install tensorflow

 

There is also the GPU package if you need it.

pip install tensorflow-gpu

 

If pip is not working for you, you can review others methods for installations here.

In your terminal check that you properly installed the library. In your terminal type the following commands and check that you don’t get any exceptions

python import tensorflow as tf hello = tf.constant(‘Hello, TensorFlow!’)sess = tf.Session()print(sess.run(hello))

If you are using PyCharm and you are dealing with issues installing Tensorflow go to Preferences => Project :_name =>Project Interpreter  select all Packages and press the upgrade button,now add a new package and type Tensorflow in the search box then hit install package button.

We will start by reviewing some key Tensorflow players so we have some basic knowledge:

Some key players….

placeholder – a value that we’ll input when we ask TensorFlow to run a computation

Variable –  in-memory buffers containing tensors.

Examples:

Creating Tensor with const values:

hello = tf.constant('Hello, TensorFlow!')

Or just a variable as float:

x = tf.Variable(4, np.float32)

 

global_variables_initializer – this step takes the initial values (in this case tensors full of zeros) that have already been specified, and assigns them to each Variable

Session The connection to this backend is called a session ,object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated. when finishing running the session make sure you hit the close method

  session.run(…)
session
.close()

Graph it contains set of the operation objects that are units f computations – nodes and the Tensor that are the edges.

Math operations such as basic arithmetic operators to your graph or tensor.

Images Tensor flow gives you methods to handle images operations such as decoding and encoding, resizing and cropping and      many more.

TensorFlow Process

We have the same general skeleton in Tensor flow application

  • create our model.
  • create a cost function
  • select an optimizer to optimze our model with our selected cost function
  • run our train data against our model, for each input try and adjust our model using the prediction of our model and the actual result that we know for our train data

Our first prediction…

We are finally up and running, we tested our basic tensorFlow code so we can now continue to some actual tasks. In this post we recognise handwritten digit using the RandomForest algorithm, we can now try and replica this experiment using the tensorFlow package using different models.

We can start with a simple example of building a simple linear regression, we can simply generate our X, Y using a known function and then see if Tensorflow will be able to recreate the wights of our function which will be:  Y=  2*x + 1.5

x = np.random.rand(1000).astype(np.float32)
y = x*2 +1.5

 

Our data samples are X – we randomly generated it  and our observed values are Y that we calculated. We will start with our function Ax + B and try to fit  the A,B weights to our knows weights( A=2,B=1.5) using a linear regression model. We will create random weights to start with, and then we will try to move to the optimal weights using a Tensorflow optimizer, we will give the optimizer a loss function to optimize and we will start with mean error loss = ( Y_predicted – Y_actual )^2

our_loss_function = tf.reduce_mean(tf.square(y - y_samples))

We will use the gradient Descent optimizer to adjust our weights and feed it with our generated samples and ask it to minimize our loss function.

optimize = tf.train.GradientDescentOptimizer(0.5)
#0.5 is the learning rate - how fast will we move to the goal
train = optimize.minimize(our_loss_function)

Now that our model is set, it’s time to shift tensorFlow into gear! we need to create our session, init the graph and then run it with our problem

init = tf.global_variables_initializer()sess = tf.Session() #Launch the graph in a session. sess.run(init) #Evaluate the tensor

Now we will run our function trying to improve it each time by adjusting the weights , let`s say we will adjust our weights  a 1000 times.

for step in range(1000):
    session.run(train)
print(session.run(weights), session.run(intercept))

*Note that session.run(our_tensor) Evaluate the tensor

The result is (array([ 1.99999917], dtype=float32), array([ 1.50000048], dtype=float32))

not bad…. the entire code is here in this github repository.

We completed our first prediction using a simple gradient decent. We can try and try our hand written digit problem that we handed last time using random forest.

Neural Network using TensorFlow

We will try our hand written digit again and see if we can score a better accuracy then we did using Random forest. As before we will import the digits that are pre categorised for us.

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

Few remarks:

  • One hot encoding technique is used to encode categorical integer features using a one-hot aka one-of-K scheme.
  • Run this 2 lines first because downloading the images takes some time.
  • Each image is 28 pixels by 28 pixels.

Congratulation! You are now a new owner of more then Thousands images of digits! We will start with reviewing an image to understand what we are looking at.

image = mnist.train.next_batch(1)
tmp = mnist.train.images[10]
tmp = tmp.reshape((28,28))
plt.imshow(tmp)

 

We try to visualze the 10th image in the repository, the result is:

mnist digit 0
figure 3

 

For us humans it seems a real easy task to notice that we have 3 and 0 as digit in the images, but we need to teach our computer to see the digits in these images.

Let`s try and build simple Neural net on this problem.

Neural net

Our basic NN will look as the following:

 

first_nn
first_nn

 We use the pixle data as the input layer so it’s 28X28=784 nodes. we then move the data to the next layer, then we move it to a softmax layer to create a distribution to our output layer that will try and predict which digit is the current number we feed the system, between the layers we have weights that we will adjust as we train our model till we are good enough.

First layer – is the input layer, we take our 784 pixels(28X28) and create a tensor out of them, we will create a placeholder that will hold the pixels as an input.

x = tf.placeholder(tf.float32, [None, 784]) # None - unlimited number of images

Our first layer is done!

Our layer is connected to the next layer with edges that will represent the weights, so we will need a variable for the weights, we can init it with 0, same goes to our bias

weights= tf.Variable(tf.zeros([784, 10]))
bias = tf.Variable(tf.zeros([10]))

[784, 10] – is our input data 784 nodes 10 classes 0-9

[10] - is our 10 nodes n the layer that will be transform to the classes using the softmax function.

We have our tensor representation of the layers so we can now move to the model.

Out main calculation is taking each pixel from the image multiply it with the corresponding weight add the bias and then apply our Softmax function to convert the values to distribution between 0-9. Because we are multiplying matrixes we will need to use the tf,matmul function as for x*weights
y = tf.nn.softmax(tf.matmul(x, weights) + bias)

We already have our Tensor and the Flows between them, our main goal is to adjust the weights so we can predict what each image represents 0-9, we will  do so by feeding our net sample by sample trying to minimize a cost function that will panish us when we are wrong, till we get good results or we are out of training data, then we will test our module using our test data.
As for the cost function, last time we used less squeare erros, this time we will use the cross enrtropy function which is just the negative sum of the true value of the probabilty(100%) and the log of our predicted value which move in the range 0-100%

Cross Entropy = – sum (Yi true_value * log (Yi pred_value))

We need to create the real value vector

y_actual = tf.placeholder(tf.float32, [None, 10])

So our function will be :

optimize_function = tf.reduce_mean(-tf.reduce_sum(y_actual * tf.log(y_predict)))

 

we will use the gradient decent to adjust our weights for each training data with steps of 0.5.
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(optimize_function)

 

0.5 - is our learning rate how much we adjust  our weights to the direction that the gradient decent tells us, the learning rate is an adjustment how fast we want to move or how accurate we would like to be.

So we minimize our cross entropy function using a gradient descent with a learning rate 0.5 you can try to adjust the learning rate and see the differences.

TensorFlow is lazy so it will not run till it must, we still didn’t enforce it to run so its time to create our session and run our tensor flow just as before

init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)

 

Finally we are ready to optimize our function using Tensorflow and check which learning rate will give us the best results

Here is the complete code:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

from PIL import Image

import matplotlib.image as mpimg

session = tf.Session()

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
image = mnist.train.next_batch(1)

tmp = mnist.train.images[1]
tmp = tmp.reshape((28, 28))

plt.imshow(tmp)

x_input = tf.placeholder(tf.float32, [None, 784])

weights = tf.Variable(tf.zeros([784, 10]))
bias = tf.Variable(tf.zeros([10]))

y_predicted = tf.nn.softmax(tf.matmul(x_input, weights) + bias)

y_actual = tf.placeholder(tf.float32, [None, 10])

optimize_function = tf.reduce_mean(-tf.reduce_sum(y_actual * tf.log(y_predicted), reduction_indices=[1]))

init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
for step in [0.01, 0.1, 0.5, 1, 2]:
    train_step = tf.train.GradientDescentOptimizer(step).minimize(optimize_function)
    for i in range(5000):
        batch_xs, batch_ys = mnist.train.next_batch(100)
        sess.run(train_step, feed_dict={x_input: batch_xs, y_actual: batch_ys})

    correct_prediction = tf.equal(tf.argmax(y_predicted, 1), tf.argmax(y_actual, 1))

    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    print(sess.run(accuracy, feed_dict={x_input: mnist.test.images, y_actual: mnist.test.labels}))

The results we are getting are:
0.9013
0.9217
0.923
0.9178
0.878

Which means that in the best case for learning rate of 0.5 we get 92.3 accuracy on our test data, not bad but we did much better in our
random forest algorithm
where we scored almost 94% accuracy, we can do much much better if we start improving our neural net providing it more hidden layers.

Convolutionals layer:

 

We will start and write our code as if we have a blackbox that we will feed it with the pixels of the images and it will predict which digit is in the image

magic_network
magic_network

We have the same usual steps in this Tensorflow application as before

  • create our model.
  • create a cost function
  • select an optimizer to optimze our model with our selected cost function
  • run our train data against our model, for each input try and adjust our model using the prediction of our model and the actual result that we know for our train data

For now we can ignore what will be happing in our blackbox (The actual neural network ) and just create the skeleton for our application

import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Image details
image_width = 28
image_height = 28
total_pixels = image_width * image_height  # # img shape: 28*28
digit_classes = 10  # 0-9 digits

 

*Each Image is 8px*8px

# Create placeholder for the input unknown size of images and 28*28 pixels
x = tf.placeholder(tf.float32, [None, total_pixels])
y = tf.placeholder(tf.float32, [None, digit_classes])

 

We create 2 place holders for our model x,y

Our model

prediction = magic_net(input)

Define our cost function – we will use the cost entropy function

cost_function = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))

learning_rate = 0.001
# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# Evaluate model
correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

 

  • The tf.train.AdamOptimizer uses Kingma and Ba’s Adam algorithm to control the learning rate. Adam offers several advantages over the simple tf.train.GradientDescentOptimizer. Foremost is that it uses moving averages of the parameters (momentum).

Now we can run on our training data and feed it to our optimizer.

init = tf.initialize_all_variables()

iters = 200000
batch_size = 100
display_step = 10

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    step = 1
    # Keep training until reach max iterations
    while step * batch_size < iters:
        batch_x, batch_y = mnist.train.next_batch(batch_size)

        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
        if step % display_step == 0:
            # Calculate batch loss and accuracy
            loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                              y: batch_y})
            print("Iter " + str(step * batch_size) + ", Minibatch Loss= " + \
                  "{:.6f}".format(loss) + ", Training Accuracy= " + \
                  "{:.5f}".format(acc))
        step += 1


 

So we have our skeleton written but nothing is really solving the problem yet… This is where we need to roll up our sleeves and code the model, we will use a technic called convolution, We create a set filters(kernels), basically think of it as a small sliding window much smaller then your image and you slide it through the image each step filtering the current part of the image using a dot product – just multiply the image pixel against the entries of the filter and producing a new image that is filtered , the result of this opperation lets the network to detect some specific features in spatial position in the image.

Remarks:

  • Kernel – is a small matrix. It is useful for blurring, sharpening, embossing, edge detection, and more. This is accomplished by means of convolution between a kernel and an image.
  • dot product – an algebraic operation that takes two equal-length sequences of numbers (usually coordinate vectors) and returns a single number
  • Stride is how much we move in the axes each time we slide the window

The next step is the polling step

Pooling step – we make each filtered image smaller by a factor

  1. pick a window
  2. set the stride
  3. walk the window pixel by pixel
  4. each window we pick the maximum value for the next layer
  5. change negative values to 0

After we finish pooling we have a new array of pixels which is  much smaller. We can now define our convolution and pulling functions, we will need to provide tensor flow with the weights that sits on the edges.

def conv2d(x, w, b, strides=1):
    # Conv2D wrapper, with bias and relu activation
    x = tf.nn.conv2d(x, w, strides=[1, strides, strides, 1], padding='SAME')
    x = tf.nn.bias_add(x, b)
    return tf.nn.relu(x)


def maxpool2d(x, k=2):
    # MaxPool2D wrapper
    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
                          padding='SAME')


def conv_pull(x, w, b, strides=1):
    conv = conv2d(x, w, b)

    # Max Pooling (down-sampling)
    conv = maxpool2d(conv, k=2)
    return conv

 

  • relu is the activation function   {\displaystyle f(x)=\max(0,x),} this is what our neuron will do

 

We will need to create a function to init our weights and bias variables – this will be our init values from there on we will try to adjust them till we get good results( this is the job of the optimizer)

def get_weights(shape):

    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))


def get_bias(length):
    return tf.Variable(tf.random_normal([length]))

 

A

TensorShape
represents a possibly-partial shape specification for a Tensor. It may be one of the following:

  • Fully-known shape: has a known number of dimensions and a known size for each dimension.
  • Partially-known shape: has a known number of dimensions, and an unknown size for one or more dimension.
  • Unknown shape: has an unknown number of dimensions, and an unknown size in all dimensions.

Its time to create our actual model aka the magic net:

We will start by reshaping each image into a 4 diminution tensor as the convolution api wants

def magic_net(x):
    # Reshape input image to a 4 tensor
    x = tf.reshape(x, shape=[-1, 28, 28, 1])

 

  • if one component of `shape` is the special value -1, the size of that dimension
    is computed so that the total size remains constant. In particular, a `shape`
    of `[-1]` flattens into 1-D. At most one component of `shape` can be -1.
  • 28- the pixel size of the image , 1 is the color channel
  • A channel in this context is the grayscale image of the same size as a color image, made of just one of these primary colors. For instance, an image from a standard digital camera will have a red, green and blue channel. A grayscale image has just one channel.

 

Now we create our first convolution layer, basically we will ask it to use a 5X5 filter on 1 image and to generate 32 images as an out put

# We would like a filter 5*5 , input is 1 image and output is 32 images that where filtered
shape = [5, 5, 1, 32]
# 1 Convolution Layer
conv1 = conv_pull(x, get_weights(shape=shape), get_bias(length=32))

 

 

Our second layer takes a filter again 5X5 this time it gets 32 images and outputs 64 images

# We would like a filter 5*5 , input is 32 images and output is 64 images that where filtered
shape = [5, 5, 32, 64]
#
# 2 Convolution Layer
conv2 = conv_pull(conv1, get_weights(shape=shape), get_bias(length=64))

 

 

 

We now have 64 images of the size 7*7 ( we started with 28 pooled it to 14 and pooled again to 7), we would like to convert the 64 images to one layer of the size 1024  and then to a 10 classes layer once for each digit. so we will create 2 layers one is 3136 downgraded to 1024 and the second is 10

# Fully connected layer we move from 64 images size 7*7 *64 = 3136
# Reshape conv2 output to fit fully connected layer input

fully_connected_length_start = 7*7*64

fully_connected_length_end = 1024

w = get_weights([fully_connected_length_start, fully_connected_length_end])
b = get_bias(fully_connected_length_end)
#we need to move from 4 dim representing the convolution  to 2 dim representing nodes
fc1 = tf.reshape(conv2, [-1, w.get_shape().as_list()[0]])
fc1 = tf.add(tf.matmul(fc1, w), b)
# convert negative to zero
fc1 = tf.nn.relu(fc1)

# Output, class prediction last layer is the prediction of the 10 classes
out = tf.add(tf.matmul(fc1, get_weights([fully_connected_length_end, digit_classes])), get_bias(digit_classes))
return out

We finally have our model ready we can run our code and review the results, you can play with the number or iterations and rate for the optimizer to see different results but our code will make an amazing result of 99% accuracy!
References and great material to go over:

https://www.tensorflow.org/tutorials/mnist/pros/

https://www.youtube.com/user/hvasslabs

https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/02_Convolutional_Neural_Network.ipynb

https://www.youtube.com/channel/UCWN3xxRkmTPmbKwht9FuE5A

 

 

 

 

First steps with TensorFlow using Python

Yoni Amishav


Tech lead, blogger, node js Angular and more ...


Post navigation


Leave a Reply

Free Email Updates
Get the latest content first.
We respect your privacy.
%d