I am studying to Recommendation System. I use to Random Forest with Tensorflow. I have a problem with my loss result. How to fix my code. Help me.
This is value of x_data
shape=(6000,116)
value is 0 or 1
array([[1, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 1, 0, 0], [0, 0, 0, ..., 0, 0, 0], ..., [0, 0, 0, ..., 1, 1, 0], [0, 0, 0, ..., 0, 0, 1], [0, 0, 0, ..., 0, 0, 1]]) This is value of y_data
shape=(6000,1)
value is 0 or 1
array([[0], [0], [1], ..., [0], [0], [0]]) This is my code
def next_batch(x_data, y_data, batch_size): if (len(x_data) != len(y_data)): return None, None batch_mask = np.random.choice(len(x_data), batch_size) x_batch = x_data[batch_mask] y_batch = y_data[batch_mask] return x_batch, y_batch x_train = train.iloc[:, 3:].values y_train = train.iloc[:,2:3].values x_test = test.iloc[:,2:].values x_data = np.array(x_train, dtype=np.float32) y_data = np.array(y_train, dtype=np.int64) test_data = np.array(x_test, dtype=np.float32) # Parameters num_steps = 500 batch_size = 1024 num_classes = 2 num_features = 116 num_trees = 10 max_nodes = 1000 tf.reset_default_graph() # Input and Target placeholders X = tf.placeholder(tf.float32, shape=[None, num_features]) Y = tf.placeholder(tf.int64, shape=[None,1]) # Random Forest Parameters hparams = tensor_forest.ForestHParams(num_classes=num_classes, num_features=num_features, num_trees=num_trees, max_nodes=max_nodes).fill() #Build the Random Forest forest_graph = tensor_forest.RandomForestGraphs(hparams) # Get training graph and loss train_op = forest_graph.training_graph(X, Y) loss_op = forest_graph.training_loss(X,Y) # Measure the accuracy infer_op, _, _ = forest_graph.inference_graph(X) correct_prediction = tf.equal(tf.argmax(infer_op, 1), tf.cast(Y, tf.int64)) accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) init_vars = tf.group(tf.global_variables_initializer(), resources.initialize_resources(resources.shared_resources())) sess = tf.Session() sess.run(init_vars) # Training for i in range(1, num_steps + 1): # Prepare Data # Get the next batch of MNIST data (only images are needed, not labels) batch_x, batch_y = next_batch(x_data, y_data, batch_size) _, l = sess.run([train_op, loss_op], feed_dict={X: batch_x, Y: batch_y}) if i % 50 == 0 or i == 1: acc = sess.run(accuracy_op, feed_dict={X: batch_x, Y: batch_y}) print('Step %i, Loss: %f, Acc: %f' % (i, l, acc)) why is my loss function return negative values?
Result
INFO:tensorflow:Constructing forest with params = INFO:tensorflow:{'num_trees': 10, 'max_nodes': 1000, 'bagging_fraction': 1.0, 'feature_bagging_fraction': 1.0, 'num_splits_to_consider': 10, 'max_fertile_nodes': 0, 'split_after_samples': 250, 'valid_leaf_threshold': 1, 'dominate_method': 'bootstrap', 'dominate_fraction': 0.99, 'model_name': 'all_dense', 'split_finish_name': 'basic', 'split_pruning_name': 'none', 'collate_examples': False, 'checkpoint_stats': False, 'use_running_stats_method': False, 'initialize_average_splits': False, 'inference_tree_paths': False, 'param_file': None, 'split_name': 'less_or_equal', 'early_finish_check_every_samples': 0, 'prune_every_samples': 0, 'num_classes': 2, 'num_features': 116, 'bagged_num_features': 116, 'bagged_features': None, 'regression': False, 'num_outputs': 1, 'num_output_columns': 3, 'base_random_seed': 0, 'leaf_model_type': 0, 'stats_model_type': 0, 'finish_type': 0, 'pruning_type': 0, 'split_type': 0} WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/tensor_forest/python/tensor_forest.py:529: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Deprecated in favor of operator or tf.math.divide. Step 1, Loss: -1.000000, Acc: 0.873047 Step 50, Loss: -250.399994, Acc: 0.833313 Step 100, Loss: -537.200012, Acc: 0.856388 Step 150, Loss: -822.799988, Acc: 0.841568 Step 200, Loss: -1001.000000, Acc: 0.835522 Step 250, Loss: -1001.000000, Acc: 0.839737 Step 300, Loss: -1001.000000, Acc: 0.817566 Step 350, Loss: -1001.000000, Acc: 0.816372 Step 400, Loss: -1001.000000, Acc: 0.843414 Step 450, Loss: -1001.000000, Acc: 0.829651 Step 500, Loss: -1001.000000, Acc: 0.839970 1 Answer
The loss is just a scalar that you are trying to minimize. It's not supposed to be positive.
One of the reason you are getting negative values in loss is because the training_loss in RandomForestGraphs is implemented using cross entropy loss or negative log liklihood as per the reference code here.
Also, as you can see the loss remains constant in the later iterations, I suppose doing Hyperparameter Tuning will make the tree robust to variances of the data. You can reference some ideas from here.
