Writing more than 28 GiB leads to failures.

[dvoytik]

Writing more than 28 GiB sequentially leads to failures.

Always successful:

[root@ndp-nvm-test ~]# dd if=/dev/zero of=/dev/nvme0n1p1 bs=1M count=29500 conv=fdatasync,notrunc
status=progress
30699159552 bytes (31 GB, 29 GiB) copied, 78.0004 s, 394 MB/s
29500+0 records in
29500+0 records out
30932992000 bytes (31 GB, 29 GiB) copied, 101.946 s, 303 MB/s

Always fails:

dd if=/dev/zero of=/dev/nvme0n1p1 bs=1M count=30000 conv=fdatasync,notrunc status=progress
31448891392 bytes (31 GB, 29 GiB) copied, 80.0064 s, 393 MB/s
dd: fdatasync failed for '/dev/nvme0n1p1': Input/output error
dd: fsync failed for '/dev/nvme0n1p1': Input/output error

Kernel v4.12.8 dmesg:

[ 8537.431183] blk_update_request: I/O error, dev nvme0n1, sector 60634400
...
[ 8537.478674] nvme nvme0: Abort status: 0x7
[ 8553.990942] nvme nvme0: Device not ready; aborting reset
[ 8553.990958] nvme nvme0: Removing after probe failure status: -19
[ 8554.047016] nvme0n1: detected capacity change from 259960864768 to 0
[ 8554.047038] buffer_io_error: 1101 callbacks suppressed
[ 8554.047042] Buffer I/O error on dev nvme0n1p1, logical block 7580127, lost async page write
[ 8554.047049] blk_update_request: 28 callbacks suppressed
[ 8554.047052] blk_update_request: I/O error, dev nvme0n1, sector 60643056
[ 8554.047055] Buffer I/O error on dev nvme0n1p1, logical block 7580126, lost async page write

No errors on firmware console, only messages of resetting after the driver resets the device:

NVME CC.EN: 0
PCIe Bus Master: 0

[dvoytik]

Maybe this will help to identify possible solution.
After nvme stopped responding I see that the cpu spins here:

Thread [1] (Suspended: Signal 'SIGINT' received. Description: Interrupt.)	
	12 set_auto_rx_dma() host_lld.c:407 0x001013c0	
	11 PopFromReqQueue() low_level_scheduler.c:212 0x001054e4	
	10 ExeLowLevelReqPerDie() low_level_scheduler.c:565 0x00107190	
	9 ExeLowLevelReqPerCh() low_level_scheduler.c:1286 0x0010cd84	
	8 ExeLowLevelReq() low_level_scheduler.c:1513 0x0010ed2c	
	7 PushToReqQueue() low_level_scheduler.c:115 0x0010487c	
	6 PrePmRead() page_map.c:434 0x00113a18	
	5 LRUBufWrite() lru_buffer.c:495 0x00111ac0	
	4 handle_nvme_io_write() nvme_io_cmd.c:123 0x001032bc	
	3 handle_nvme_io_cmd() nvme_io_cmd.c:148 0x00103374	
	2 nvme_main() nvme_main.c:136 0x00103570	
	1 main() main.c:129 0x0011210c	

void set_auto_rx_dma(unsigned int cmdSlotTag, unsigned int cmd4KBOffset, unsigned int devAddr)
{
	/* ... */

	while((g_hostDmaStatus.fifoTail.autoDmaRx + 1) % 256 == g_hostDmaStatus.fifoHead.autoDmaRx)
		g_hostDmaStatus.fifoHead.dword = IO_READ32(HOST_DMA_FIFO_CNT_REG_ADDR);

        /* ... */
}

[dvoytik]

Accordingly for a read test:

dd if=/dev/nvme0n1 of=/dev/null bs=1M count=204800 status=progress

CPU stack:

Thread [1] (Suspended: Signal 'SIGINT' received. Description: Interrupt.)	
	7 set_auto_tx_dma() host_lld.c:377 0x001011c0	
	6 PopFromReqQueue() low_level_scheduler.c:228 0x00105810	
	5 ExeLowLevelReqPerDie() low_level_scheduler.c:560 0x00107158	
	4 ExeLowLevelReqPerCh() low_level_scheduler.c:1281 0x0010cd4c	
	3 ExeLowLevelReq() low_level_scheduler.c:1508 0x0010ecf4	
	2 nvme_main() nvme_main.c:193 0x001037e8	
	1 main() main.c:129 0x00112068	

Spins infinitely here:

void set_auto_tx_dma(unsigned int cmdSlotTag, unsigned int cmd4KBOffset, unsigned int devAddr)
{
	/* ... */
	while((g_hostDmaStatus.fifoTail.autoDmaTx + 1) % 256 == g_hostDmaStatus.fifoHead.autoDmaTx)
		g_hostDmaStatus.fifoHead.dword = IO_READ32(HOST_DMA_FIFO_CNT_REG_ADDR);
	/* ... */
}

[dvoytik]

Could it be that, because of overflow, HOST_DMA_FIFO_CNT_REG_ADDR register stops updating? If so then maybe there is some kind of an error flag which I can read and check?

[Sangjin-Lee]

Greedy FTL does not check the status of the host DMA command fifo before enqueuing host DMA commands to the fifo, so a sufficiently large nlb of NVMe IO command can cause problems.

Could you check 'nlb' of NVMe IO command while dd (with bs=1M) is running?

[dvoytik]

Hi @Sangjin-Lee,

if dd bs=256k or more then in GreedyFTL nlb >= MAX_NUM_OF_NLB

dd bs=257K  --> NLB = 129
dd bs=508K  --> NLB = 253
dd bs=510K  --> NLB = 255
dd bs=512K  --> NLB = 255
dd bs=1M    --> NLB = 255

[dvoytik]

I see the same problem with 4K block size:

dd if=/dev/nvme0n1 of=/dev/null bs=4K count=8126464 status=progress
31257419776 bytes (31 GB, 29 GiB) copied, 16 s, 2.0 GB/

and it stucks.

[Cosmos-OpenSSD]

This phenomenon could not be reproduced in our development environment
If additional experimentation reproduces this issue, I'll give you additional answers.