uniform insert full

[KINGFIOX]

When I inserted the pea hash with uniform distribution in dram unique edition, I found it was blocked as it reach 230.1M data. After waiting for a long time, I chose to interrupt the program.

This is the logs:

No.1 insert pea 1threads test begin
Distribution = uniform
EPOCH registration in application level
Pea
Comprehensive Benchmark
insertion start
Insert1
(below is the logs printings I added in the code)
Inserted 10000000 entries so far. Throughput: 5.31116e+06 entries per second.
Inserted 20000000 entries so far. Throughput: 3.26115e+06 entries per second.
Inserted 30000000 entries so far. Throughput: 5.72216e+06 entries per second.
Inserted 40000000 entries so far. Throughput: 4.639e+06 entries per second.
Inserted 50000000 entries so far. Throughput: 5.27367e+06 entries per second.
Inserted 60000000 entries so far. Throughput: 1.35598e+06 entries per second.
Inserted 70000000 entries so far. Throughput: 6.79158e+06 entries per second.
Inserted 80000000 entries so far. Throughput: 6.15792e+06 entries per second.
Inserted 90000000 entries so far. Throughput: 5.62764e+06 entries per second.
Inserted 100000000 entries so far. Throughput: 5.03531e+06 entries per second.
Inserted 110000000 entries so far. Throughput: 4.54232e+06 entries per second.
Inserted 120000000 entries so far. Throughput: 5.15846e+06 entries per second.
Inserted 130000000 entries so far. Throughput: 5.12154e+06 entries per second.
Inserted 140000000 entries so far. Throughput: 5.13705e+06 entries per second.
Inserted 150000000 entries so far. Throughput: 5.09772e+06 entries per second.
Inserted 160000000 entries so far. Throughput: 5.10169e+06 entries per second.
Inserted 170000000 entries so far. Throughput: 5.078e+06 entries per second.
Inserted 180000000 entries so far. Throughput: 4.84972e+06 entries per second.
Inserted 190000000 entries so far. Throughput: 4.68886e+06 entries per second.
Inserted 200000000 entries so far. Throughput: 4.59412e+06 entries per second.
Inserted 210000000 entries so far. Throughput: 4.47617e+06 entries per second.
Inserted 220000000 entries so far. Throughput: 4.28257e+06 entries per second.
Inserted 230000000 entries so far. Throughput: 1.11766e+06 entries per second.

[KINGFIOX]

同样的，当我使用unique、skew=0进行插入的时候，也是插到 270M 的时候，就差不进去了。其他的 pre-load 都是默认值。

[KINGFIOX]

running args: ./test_pmem -distribution uniform -index pea -p 500000000 -t 1 -op insert

[DrWereviruswolf]

For bulkloading 270M key-value pairs, you should change the following code:

1. Set the size of memory pool > 16GB:

   peahash/pea_dram/unique/test/test_pmem.cpp

   Line 56 in a489567

   DEFINE_uint64(ps, 10ul, "The size of the memory pool (GB)");

   
   I added the check of memory pool in the latest commit.
2. Quadruple the size of lock table, which should be 589824 instead of
   

   peahash/pea_dram/unique/src/PeaHash/pea_hash.h

   Line 22 in a489567

   #define LOCK_TABLE_SIZE 147456

   
   Here I use a simple way to implement the lock table, which should have been re-allocated as the directory doubling. If I have free time, I will modify the code of lock table resizing.

[KINGFIOX]

但是有点奇怪的就是，skew=0.8（也就是默认值），确实是可以插入 500M 以上的数据

[DrWereviruswolf]

unique版本下那个skew参数没有用。duplicate版本下插入500M skew=0.8时，实际空间占用可能是够的，因为大量相同key的value被压缩进了value segment(bucket)。

[DrWereviruswolf]

Lock table resizing is implemented in a specific branch dev and merged into main branch. pea_pmem has the same issue to fix.

peahash/pea_dram/unique/src/PeaHash/pea_hash.h

Line 880 in 310620d

void *new_lock_mem = realloc(lock_map_table, lock_table_size * sizeof(lock_type));

.