Streaming Reasoning

[mcmingchang]

I tried to add a new interface for v1/audio/speech-stream to the AI. The process worked, but there was a current sound at the junction of each returned audio data packet. I hope the official version will be released soon

[Lourdle]

I understand. There are still a few minor bugs on my side, and I’ve already fixed them. I’ll publish a release first, then start working on streaming support.
I haven’t begun that work yet. If you’ve already implemented streaming support and it works but still has some issues, please share the code showing the feature or open a PR. That would help me move faster.

[Lourdle]

I've added initial streaming TTS support in PR #23. Note that this is still a work-in-progress draft; stability especially with longer inputs is being improved. Please follow PR #23 for progress and updates.

[mcmingchang]

谢谢大佬，流式推理是合并到main了吗，是server端的流式推理吗，tool_zh.md还没看到有调用方法

[Lourdle]

还没呢，我还在优化性能。目前只有 cli 的交互模式支持这个，server 的流式支持会在优化后添加，文档也会在全部完成后更新。可以跟踪 PR #23 查看进度。

[mcmingchang]

我打算等流式出来后把http升级为websocket，配合funasr就可以接agent做到实时对话啦

[mcmingchang]

我让AI部署了你的流式代码，10个字短文本是没问题得，但是50+的长文本会报错，我把报错信息贴出来看看你能不能修复

========================================
FATAL: Caught signal 11 (SIGSEGV)
Backtrace (13 frames):

./build-cuda/bin/cosyvoice-server(+0xf9be)[0x638e762f39be]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7b5cb7c42520]
/home/csrobot/chenmc/cosyvoice.cpp/build-cuda/bin/../lib/libcosyvoice.so(+0x120e51)[0x7b5cb8520e51]
/home/csrobot/chenmc/cosyvoice.cpp/build-cuda/bin/../lib/libcosyvoice.so(cosyvoice_tts_instruct_stream+0x1a)[0x7b5cb851e01a]
./build-cuda/bin/cosyvoice-server(+0x1b8a6)[0x638e762ff8a6]
./build-cuda/bin/cosyvoice-server(+0x29f43)[0x638e7630df43]
./build-cuda/bin/cosyvoice-server(+0x51d0b)[0x638e76335d0b]
./build-cuda/bin/cosyvoice-server(+0x7571a)[0x638e7635971a]
./build-cuda/bin/cosyvoice-server(+0x762de)[0x638e7635a2de]
./build-cuda/bin/cosyvoice-server(+0x3c3ae)[0x638e763203ae]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xf6ca6)[0x7b5cb80f6ca6]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7b5cb7c94ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x1268d0)[0x7b5cb7d268d0]

(py310) csrobot@csrobot-DGX-Station:~/chenmc/cosyvoice.cpp$ nm -C /home/csrobot/chenmc/cosyvoice.cpp/build-cuda/lib/libcosyvoice.so | grep -E "^0+11[ef]|^0+120[0-9a-f]|^0+121[0-9a-f]|^0+122[0-9a-f]" | sort
000000000011e000 T cosyvoice_tts_instruct_stream
000000000011e020 T cosyvoice_tts_cross_lingual_stream
000000000011e040 T cosyvoice_tts_context_new
000000000011e600 t std::_Function_handler<unsigned long (std::basic_string_view<char, std::char_traits >), cosyvoice_tts_context::tts_job(char const*, char const*, float, cosyvoice_inference_mode, cosyvoice_generated_speech*, bool ()(float const, unsigned int, void*), void*)::{lambda(std::basic_string_view<char, std::char_traits >)#1}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation)
000000000011e640 t cosyvoice_tts_context::cosyvoice_tts_stream_with_postprocess(int const*, unsigned int, float, bool ()(float const, unsigned int, void*), void*)::{lambda(float const*, unsigned int, void*)#1}::_FUN(float const*, unsigned int, void*)
000000000011e860 t std::_Function_handler<unsigned long (std::basic_string_view<char, std::char_traits >), cosyvoice_tts_context::tts_job(char const*, char const*, float, cosyvoice_inference_mode, cosyvoice_generated_speech*, bool ()(float const, unsigned int, void*), void*)::{lambda(std::basic_string_view<char, std::char_traits >)#1}>::_M_invoke(std::_Any_data const&, std::basic_string_view<char, std::char_traits >&&)
000000000011e8e0 t std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > >::~vector()
000000000011e8e0 t std::vector<std::vector<int, std::allocator >, std::allocator<std::vector<int, std::allocator > > >::~vector()
000000000011e960 t cosyvoice_tts_context::tts_job(char const*, char const*, float, cosyvoice_inference_mode, cosyvoice_generated_speech*, bool ()(float const, unsigned int, void*), void*)
0000000000122290 T cosyvoice_log_callback_default
0000000000122330 t use_count_guard::use_count_guard(cosyvoice_model*)
0000000000122330 t use_count_guard::use_count_guard(cosyvoice_model*)
0000000000122350 t use_count_guard::use_count_guard(cosyvoice_context*)
0000000000122350 t use_count_guard::use_count_guard(cosyvoice_context*)
0000000000122390 t use_count_guard::~use_count_guard()
0000000000122390 t use_count_guard::~use_count_guard()
00000000001223c0 T cosyvoice_init_backend
0000000000122410 T cosyvoice_init_backend_from_path
0000000000122430 T cosyvoice_duplicate_context
0000000000122510 T cosyvoice_free
0000000000122530 T cosyvoice_llm_prefill
0000000000122550 T cosyvoice_llm_decode
0000000000122570 T cosyvoice_llm_prepare_probs
0000000000122590 T cosyvoice_llm_get_kv_cache_len
00000000001225b0 T cosyvoice_llm_set_kv_cache_len
00000000001225d0 T cosyvoice_llm_offload_kv_cache
00000000001225f0 T cosyvoice_llm_load_kv_cache
0000000000122610 T cosyvoice_llm_sample_token
0000000000122630 T cosyvoice_llm_is_stop_token
0000000000122650 T cosyvoice_llm_accept_token
0000000000122670 T cosyvoice_llm_clear_accepted_tokens
0000000000122690 T cosyvoice_llm_get_n_accepted_tokens
00000000001226b0 T cosyvoice_llm_get_accepted_tokens
00000000001226d0 T cosyvoice_llm_job
00000000001226f0 T cosyvoice_llm_job_ext
0000000000122710 T cosyvoice_token2wav
0000000000122730 T cosyvoice_token2wav_ext
0000000000122760 T cosyvoice_tts
0000000000122950 T cosyvoice_tts_stream
0000000000122ca0 T cosyvoice_get_chunk_tokens
0000000000122cc0 T cosyvoice_set_chunk_tokens
0000000000122ce0 T cosyvoice_get_last_status
0000000000122d00 T cosyvoice_get_memory_usage
0000000000122d20 T cosyvoice_get_total_memory_usage
0000000000122d40 T cosyvoice_empty_buffer_cache
0000000000122d60 T cosyvoice_request_stop
0000000000122d80 T cosyvoice_stop_requested
0000000000122da0 T cosyvoice_set_noise_callback
0000000000122dc0 T cosyvoice_get_noise_callback
0000000000122de0 T cosyvoice_get_hift_rand_ini_len
0000000000122e00 T cosyvoice_set_hift_rand_ini
0000000000122e20 T cosyvoice_get_word_token_embed_weight
0000000000122e40 T cosyvoice_get_speech_token_embed_weight
0000000000122e60 T cosyvoice_tokenize
0000000000122eb0 T cosyvoice_tokenize_ext
0000000000122ec0 T cosyvoice_tokenizer_free
0000000000122f50 T cosyvoice_get_tokenizer
0000000000122f80 T cosyvoice_get_default_generation_config
0000000000122fa0 T cosyvoice_get_generation_config
0000000000122fc0 T cosyvoice_set_generation_config
0000000000122fe0 T cosyvoice_get_sample_rate

[Lourdle]

我刚刚提交修复代码，你试试现在呢。