Saya memiliki sekitar 550K sampel, setiap sampel berukuran 200x50x1. Ukuran kumpulan data ini sekitar 57GB.

Saya ingin melatih jaringan di set ini tetapi saya kesulitan membacanya.

batch_size=8

def _read_py_function(filename,labels_slice):
    with h5py.File(filename, 'r') as f:
        data_slice = np.asarray(f['feats'])
        print(data_slice.shape)
    return data_slice, labels_slice

placeholder_files = tf.placeholder(tf.string, [None])
placeholder_labels = tf.placeholder(tf.int32, [None])

dataset = tf.data.Dataset.from_tensor_slices((placeholder_files,placeholder_labels))
dataset = dataset.map(
    lambda filename, label: tuple(tf.py_func(
        _read_py_function, [filename,label], [tf.uint8, tf.int32])))

dataset = dataset.shuffle(buffer_size=50000)
dataset = dataset.batch(batch_size)

iterator = tf.data.Iterator.from_structure(dataset.output_types, dataset.output_shapes)
data_X, data_y = iterator.get_next()
data_y = tf.cast(data_y, tf.int32)

net = conv_layer(inputs=data_X,num_outputs=8, kernel_size=3, stride=2, scope='rcl_0')
net = pool_layer(inputs=net,kernel_size=2,scope='pl_0')

net = dropout_layer(inputs=net,scope='dl_0')

net = flatten_layer(inputs=net,scope='flatten_0')
net = dense_layer(inputs=net,num_outputs=256,scope='dense_0')
net = dense_layer(inputs=net,num_outputs=64,scope='dense_1')
out = dense_layer(inputs=net,num_outputs=10,scope='dense_2')

Dan saya menjalankan sesi menggunakan:

sess.run(train_iterator, feed_dict = {placeholder_files: filenames, placeholder_labels: ytrain})
        try:
            while True:
                _, loss, acc = sess.run([train_op, loss_op, accuracy_op])
                train_loss += loss 
                train_accuracy += acc
        except tf.errors.OutOfRangeError:
            pass

Tetapi saya mendapatkan kesalahan bahkan sebelum menjalankan sesi:

Traceback (most recent call last):
  File "SFCC-trial-134.py", line 297, in <module>
    net = rcnn_layer(inputs=data_X,num_outputs=8, kernel_size=3, stride=2, scope='rcl_0')
  File "SFCC-trial-134.py", line 123, in rcnn_layer
    reuse=False)
  File "SFCC-trial-134.py", line 109, in conv_layer
    reuse         = reuse
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1154, in convolution2d
    conv_dims=2)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 183, in func_with_args
    return func(*args, **current_args)
  File "/home/priyam.jain/tensorflow-gpu-python3/lib/python3.5/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1025, in convolution
    (conv_dims + 2, input_rank))
TypeError: %d format: a number is required, not NoneType

Saya berpikir tentang menggunakan TFRecords tetapi kesulitan membuatnya. Tidak dapat menemukan pos yang bagus di mana saya belajar membuatnya untuk jenis dataset saya.

Conv_layer didefinisikan sebagai berikut:

def conv_layer(inputs, num_outputs, kernel_size, stride, normalizer_fn=None, activation_fn=nn.relu, trainable=True, scope='noname', reuse=False):

    net = slim.conv2d(inputs = inputs,
        num_outputs   = num_outputs,
        kernel_size   = kernel_size,
        stride        = stride,
        normalizer_fn = normalizer_fn,
        activation_fn = activation_fn,
        trainable     = trainable,
        scope         = scope,
        reuse         = reuse
        )
    return net
1
Priyam Jain 19 September 2019, 17:51

1 menjawab

Jawaban Terbaik

Jangan berikan tf.py_func di dalam fungsi peta Anda. Anda dapat membaca gambar file dengan meneruskan nama fungsi langsung di dalam fungsi map Anda. Saya hanya berpose bagian kode yang relevan.

def _read_py_function(filename, label):
    return tf.zeros((224, 224, 3), dtype=tf.float32), tf.ones((1,), dtype=tf.int32)

dataset = dataset.map(lambda filename, label: _read_py_function(filename, label))

Perubahan lainnya adalah iterator Anda hanya mengharapkan input floating point. Jadi, Anda harus mengubah jenis output tf.uint8 menjadi float.

0
Prasad 21 September 2019, 18:14