Benchmark adds four new data types: STRING BLOB TIMESTAMP DATE #433
Conversation
|
|
| # 插入数据的数据类型的比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | ||
| # INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1 | ||
| # 插入数据的数据类型的比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE | ||
| INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:1:1:1:1 |
There was a problem hiding this comment.
INSERT_DATATYPE_PROPORTION=1:1:1:1:1:1:1:1:1:1
| @@ -112,6 +112,12 @@ | |||
| <artifactId>opencsv</artifactId> | |||
| <version>5.5.2</version> | |||
| </dependency> | |||
| <dependency> | |||
There was a problem hiding this comment.
是否仅仅依赖这个 tsfile 快照版本是不够用的？因为还要更新 session 版本？如果是的话那就直接依赖 session 快照版本？
| * 插入数据的比例 Data Type, D1:D2:D3:D4:D5:D6 D1: BOOLEAN D2: INT32 D3: INT64 D4: FLOAT D5: DOUBLE D6: | ||
| * TEXT | ||
| * 插入数据的比例 Data Type, D1:D2:D3:D4:D5:D6:D7:D8:D9:D9:D10 D1: BOOLEAN D2: INT32 D3: INT64 D4: FLOAT | ||
| * D5: DOUBLE D6:TEXT D7: STRING D8: BLOB D9: TIMESTAMP D10: DATE 0:0:0:0:0:0:0:0:1:0 |
There was a problem hiding this comment.
最后这个 0:0:0:0:0:0:0:0:1:0 是啥
There was a problem hiding this comment.
已删除。之前debug是的数据类型比例
| @@ -211,6 +211,12 @@ private void loadProps() { | |||
| config.setENCODING_DOUBLE( | |||
| properties.getProperty("ENCODING_DOUBLE", config.getENCODING_DOUBLE())); | |||
| config.setENCODING_TEXT(properties.getProperty("ENCODING_TEXT", config.getENCODING_TEXT())); | |||
| config.setENCODING_STRING( | |||
| properties.getProperty("ENCODING_STRING", config.getENCODING_STRING())); | |||
| config.setENCODING_BLOB(properties.getProperty("ENCODING_BLOB", config.getENCODING_BLOB())); | |||
There was a problem hiding this comment.
对应的配置参数先加上吧，即使只有一个，万一未来支持扩展了就不用再新增 bm 参数了
| @@ -536,7 +544,8 @@ private double[] generateProbabilities(int typeNumber) { | |||
| // Origin proportion array | |||
| double[] proportions = new double[typeNumber]; | |||
| LOGGER.info( | |||
| "Init SensorTypes: BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT= {}", INSERT_DATATYPE_PROPORTION); | |||
| "Init SensorTypes: BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE= {}", | |||
There was a problem hiding this comment.
考虑兼容性，如果老版本的配置文件上来需要做特殊处理的
There was a problem hiding this comment.
在ConfigDescriptor初始化配置文件时，对 INSERT_DATATYPE_PROPORTION 进行特判。输出相应的数据类型比例。
| @@ -133,6 +135,19 @@ private static long getCurrentTimestampStatic(long stepOffset) { | |||
| return Constants.START_TIMESTAMP * timeStampConst + offset + timestamp; | |||
| } | |||
|
|
|||
| private static long generateRandomTimestamp(long startTimeMillis, long endTimeMillis) { | |||
| Random random = new Random(); | |||
There was a problem hiding this comment.
不要创建这么多临时 random 对象，可以考虑跟 dataRandom 复用起来或者使用 ThreadLocalRandom
There was a problem hiding this comment.
此处已删除，采用 Function 中 long 生成分布。
| // 从 1000-01-01 开始是 valid | ||
| LocalDate start = LocalDate.of(1000, 1, 1); | ||
| int daysBetween = (int) ChronoUnit.DAYS.between(start, end); | ||
| Random random = new Random(); |
There was a problem hiding this comment.
此处已删除，采用 Function 中 long 生成分布，然后将 int 转成 LocalDate 。
| value = new Binary(blob); | ||
| break; | ||
| case TIMESTAMP: | ||
| value = generateRandomTimestamp(0, System.currentTimeMillis()); |
There was a problem hiding this comment.
可以考虑使用跟 long 类似的生成分布，可参考 Function.getValueByFunctionIdAndParam(param, currentTimestamp);
| value = generateRandomTimestamp(0, System.currentTimeMillis()); | ||
| break; | ||
| case DATE: | ||
| value = generateRandomDate(LocalDate.now()); |
There was a problem hiding this comment.
可以考虑使用跟 long 类似的生成分布，可参考 FunctionParam 看要不要抽几种分布出来
There was a problem hiding this comment.
此处采用了 Function 中 long 的生成分布。
| Binary[] sensorsBlob = (Binary[]) values[recordValueIndex]; | ||
| sensorsBlob[recordIndex] = | ||
| binaryCache.computeIfAbsent( | ||
| String.valueOf((Binary) record.getRecordDataValue().get(recordValueIndex)), |
There was a problem hiding this comment.
这块的数据类型对齐一下，是不是前面统一都用 string 就可以了，否则对于 blob 类型转来转去
There was a problem hiding this comment.
此处已改成 String， 减少类型转换次数。
First, add four new data types "STRING BLOB TIMESTAMP DATE" in SensorType.java, and then write the data random generation function in generateworkLoad.java. Secondly, add the new type conversion in the genTablet method. Finally, set the encoding method of the new data type in the configuration file.