Fix charset conversion for concatenated deliver_sm parts - #170
Open
cl77 wants to merge 1 commit into
Open
Conversation
Parts of concatenated messages (with UDH) were sent with the internal UTF-8 representation of the message text, while single-part messages were correctly re-encoded to GSM 03.38 before being written to the PDU. As a result, ESMEs received concatenated text messages with data_coding 0 whose payload was actually UTF-8, e.g. the Euro sign arrived as e2 82 ac instead of the GSM7 escape sequence 1b 65. Apply the same re-encoding logic used for single-part messages to the UDH branch: convert DC_7BIT text parts from UTF-8 to GSM 03.38 (or to alt-charset when configured) before prepending the UDH. Binary (8-bit) and UCS-2 parts are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Parts of concatenated messages (with UDH) are written to the deliver_sm PDU using ksmppd's internal UTF-8 text representation, while single-part messages are correctly re-encoded to GSM 03.38 first.
In
smpp_pdu_msg_to_pdu()(smpp/libsmpp/smpp_pdu_util.c), the loop over the parts produced bysms_split()has two branches:short_messageis duplicated and then converted viacharset_utf8_to_gsm()/alt-charsetwhendata_codingis 0 - correct.short_message = octstr_cat(udhdata, msgdata)with no charset conversion at all - the raw UTF-8 bytes leak into the PDU.As a result, ESMEs receive concatenated text messages announced with
data_coding = 0(GSM default alphabet) whose payload is actually UTF-8. Example: the Euro sign arrives ase2 82 acinstead of the GSM7 escape sequence1b 65. Single-part messages containing the same characters arrive correctly, which makes the behaviour inconsistent and impossible for the receiving ESME to interpret reliably.Fix
Apply the same re-encoding logic already used for single-part messages to the UDH branch, before the UDH is prepended:
data_coding & 0xF0ordata_coding == 0withoutalt_charset: convert UTF-8 to GSM 03.38data_coding == 0withalt_charset: convert to the configured alternative charsetOnly
DC_7BITtext parts are converted; binary (8-bit) and UCS-2 parts are unaffected (UCS-2 already worked correctly since the internal representation is UCS-2BE for those).Verification
Test harness calling
smpp_pdu_msg_to_pdu()with a 183-character DC_7BIT message containing a Euro sign (split into 2 parts,data_coding = 0):Before (part 0, after 6-byte concat UDH):
After:
Single-part messages, UCS-2 and binary messages are unchanged.