tfrecord原理详解 手把手教生成tfrecord文件与解析tfrecord文件

1.什么是tfrecord
TFRecord 是Google官方推荐的一种数据格式，是Google专门为TensorFlow设计的一种数据格式。
TFRecord本质上是二进制文件，目的是更好的利用内存。用户可以将训练集/测试集打包成生成TFRecord文件，后续就可以配合TF中相关的API实现数据的加载，处理，训练等一系列工作，可以方便高效的训练与评估模型。
2.tfrecord原理
TFRecord 并非是TensorFlow唯一支持的数据格式，你也可以使用CSV或文本等格式，但是对于TensorFlow来说，TFRecord 是最友好也是最方便的。 tf.Example是TFRecord的基本结果，其实他就是一个Protobuffer定义的message，表示一组string到bytes value的映射。TFRecord文件里面存储的就是序列化的tf.Example。在github上tensorflow的源码就能看到其定义 message Example
message Example {
Features features = 1;
};

里面只有一个变量features。如果我们继续查看Features
message Features {
// Map from feature name to feature.
map<string, Feature> feature = 1;
};

features里面就是一组string到Feature的映射。其中这个string表示feature name，后面的Feature又是一个message
继续查看Feature的定义
message Feature {
// Each feature can be exactly one kind.
oneof kind {
BytesList bytes_list = 1;
FloatList float_list = 2;
Int64List int64_list = 3;
}
};

到这里，我们就可以看到tfrecord里存储的真正数据类型有三种 bytes_list: 可以存储string 和byte两种数据类型。 float_list: 可以存储float(float32)与double(float64) 两种数据类型 。 int64_list: 可以存储：bool, enum, int32, uint32, int64, uint64 。
3.实操生成tfrecords文件
下面来手把手教大家如何生成tfrecords文件，并解析tfrecords文件。 我们以titanic数据为例
PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked
1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S
2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C
3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S
4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S
5,0,3,"Allen, Mr. William Henry",male,35,0,0,373450,8.05,,S
6,0,3,"Moran, Mr. James",male,,0,0,330877,8.4583,,Q
7,0,1,"McCarthy, Mr. Timothy J",male,54,0,0,17463,51.8625,E46,S
8,0,3,"Palsson, Master. Gosta Leonard",male,2,3,1,349909,21.075,,S
9,1,3,"Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)",female,27,0,2,347742,11.1333,,S

上面是titanic部分数据，第一行为各列字段名，后面几行为具体数据。如果想看完整的titanic数据，大家可以自行网上搜索并下载。
首先定义几个辅助方法
import tensorflow as tf
import csv

# Generate Integer Features.
def build_int64_feature(data):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[data]))

# Generate Float Features.
def build_float_feature(data):
return tf.train.Feature(float_list=tf.train.FloatList(value=[data]))

# Generate String Features.
def build_string_feature(data):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[str(data).encode()]))

然后再定义生成Example的方法
# Generate a TF `Example`, parsing all features of the dataset.
def convert_to_tfexample(survived, pclass, name, sex, age, sibsp, parch, ticket, fare):
return tf.train.Example(
features=tf.train.Features(
feature={
'survived': build_int64_feature(survived),
'pclass': build_int64_feature(pclass),
'name': build_string_feature(name),
'sex': build_string_feature(sex),
'age': build_string_feature(age),
'sibsp': build_int64_feature(sibsp),
'parch': build_int64_feature(parch),
'ticket': build_string_feature(ticket),
'fare': build_float_feature(fare),
})
)

再将其写入文件
def write_tf_records():
writer = tf.io.TFRecordWriter('output.tfrecords')
with open('titanic.csv') as f:
reader = csv.reader(f, skipinitialspace=True)
for i, record in enumerate(reader):
if i == 0:
continue
survived, pclass, name, sex, age, sibsp, parch, ticket, fare = record[1:10]
print("age, fare is: ", age, fare)
example = convert_to_tfexample(int(survived), int(pclass), name, sex, age, int(sibsp), int(parch), ticket, float(fare))

writer.write(example.SerializeToString())

这样，就生成了名为output.tfrecords的文件。
4.解析tfrecords文件
接下来我们解析上面生成的文件
首先定义features字典：
features = {
'survived': tf.io.FixedLenFeature([], tf.int64),
'pclass': tf.io.FixedLenFeature([], tf.int64),
'name': tf.io.FixedLenFeature([], tf.string),
'sex': tf.io.FixedLenFeature([], tf.string),
'age': tf.io.FixedLenFeature([], tf.string),
'sibsp': tf.io.FixedLenFeature([], tf.int64),
'parch': tf.io.FixedLenFeature([], tf.int64),
'ticket': tf.io.FixedLenFeature([], tf.string),
'fare': tf.io.FixedLenFeature([], tf.float32)
}

然后使用parse_single_example方法，解析单条数据
# Parse features, using the above template.
def parse_record(record):
return tf.io.parse_single_example(record, features=features)

主方法：
def read_tf_records():
filenames = ["output.tfrecords"]
data = tf.data.TFRecordDataset(filenames)
data = data.map(parse_record)
data = data.repeat()
# Shuffle data.
data = data.shuffle(buffer_size=1000)
# Batch data (aggregate records together).
data = data.batch(batch_size=4)
# Prefetch batch (pre-load batch for faster consumption).
data = data.prefetch(buffer_size=1)

# Dequeue data and display.
for record in data.take(1):
print("record is: ", record)
print("record[survived is: ", record['survived'])
print(type(record['survived']))
print()
print(record['survived'].numpy())
print(record['name'].numpy())
print(record['fare'].numpy())

主方法的输出为：
record is: {'age': <tf.Tensor: shape=(4,), dtype=string, numpy=array([b'', b'9', b'20', b'32'], dtype=object)>, 'fare': <tf.Tensor: shape=(4,), dtype=float32, numpy=array([16.1 , 27.9 , 15.7417, 7.925 ], dtype=float32)>, 'name': <tf.Tensor: shape=(4,), dtype=string, numpy=
array([b'Davison, Mrs. Thomas Henry (Mary E Finck)',
b'Skoog, Miss. Mabel', b'Nakid, Mr. Sahid', b'Jussila, Mr. Eiriik'],
dtype=object)>, 'parch': <tf.Tensor: shape=(4,), dtype=int64, numpy=array([0, 2, 1, 0])>, 'pclass': <tf.Tensor: shape=(4,), dtype=int64, numpy=array([3, 3, 3, 3])>, 'sex': <tf.Tensor: shape=(4,), dtype=string, numpy=array([b'female', b'female', b'male', b'male'], dtype=object)>, 'sibsp': <tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 3, 1, 0])>, 'survived': <tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 0, 1, 1])>, 'ticket': <tf.Tensor: shape=(4,), dtype=string, numpy=array([b'386525', b'347088', b'2653', b'STON/O 2. 3101286'], dtype=object)>}
record[survived is: tf.Tensor([1 0 1 1], shape=(4,), dtype=int64)
<class 'tensorflow.python.framework.ops.EagerTensor'>

[1 0 1 1]
[b'Davison, Mrs. Thomas Henry (Mary E Finck)' b'Skoog, Miss. Mabel'
b'Nakid, Mr. Sahid' b'Jussila, Mr. Eiriik']
[16.1 27.9 15.7417 7.925 ]

使用上面的方式，就解析出来原有的数据！来源：bitcarmanlee