wget command can be downloaded, but curl command is not downloaded

[sanshengshui]

As follows:

pom.xml:

<!--ftp-->
<dependency>
  <groupId>org.apache.ftpserver</groupId>
  <artifactId>ftpserver-core</artifactId>
  <version>1.1.1</version>
</dependency>

[elecharny]

Hi,

have you tried with ftpserver 1.1.4?

[sanshengshui]

Ok, I'll try👌

By the way, what's the difference between these two versions？

[elecharny]

1.1.2: https://issues.apache.org/jira/projects/FTPSERVER/versions/12340451
1.1.3 and 1.1.4 were mainly bumping up some dependencies.

I'd just like to know if you still have the issue with a maintained version, in order to investigate it with the latest code base.

thanks!

[sanshengshui]

I just upgraded to 1.1.4
Here's what happens when you download with curl

➜  Downloads curl -u 0297XfYHmj:Cbr3QqJC -o ocpp.tar.gz ftp://f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 198.19.10.106:8183...
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
< 220 Service ready for new user.
> USER 0297XfYHmj
< 331 User name okay, need password for 0297XfYHmj.
> PASS Cbr3QqJC
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Passive Mode (|||13005|)
*   Trying 198.19.10.106:13005...
* Connecting to 198.19.10.106 (198.19.10.106) port 13005
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
> TYPE I
< 200 Command TYPE okay.
> SIZE 1698304726c2bcfc8265c.tar.gz
< 213 16483570
> RETR 1698304726c2bcfc8265c.tar.gz
  0 15.7M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< 150 File status okay; about to open data connection.
* Maxdownload = -1
* Getting file with size: 16483570
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0{ [0 bytes data]
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0* transfer closed with 16483570 bytes remaining to read
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
* Closing connection 0
curl: (18) transfer closed with 16483570 bytes remaining to read

It's a different phenomenon than before，What are the possible reasons?

[elecharny]

Can you check if you have any log on the FtpServer side?

More specifically, any log containing the string "Exception during data transfer, closing data connection socket".

[sanshengshui]

The error message is as follows:

2023-10-26 16:41:01.018|TID:N/A|WARN |org.apache.ftpserver.impl.IODataConnectionFactory|createDataSocket|351|pool-18-thread-594| FtpDataConnection.getDataSocket()
java.net.SocketTimeoutException: Accept timed out
        at java.base/java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:458)
        at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:565)
        at java.base/java.net.ServerSocket.accept(ServerSocket.java:533)
        at org.apache.ftpserver.impl.IODataConnectionFactory.createDataSocket(IODataConnectionFactory.java:329)
        at org.apache.ftpserver.impl.IODataConnectionFactory.openConnection(IODataConnectionFactory.java:231)
        at com.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand.execute(RetrCommand.java:74)
        at org.apache.ftpserver.impl.DefaultFtpHandler.messageReceived(DefaultFtpHandler.java:211)
        at org.apache.ftpserver.listener.nio.FtpHandlerAdapter.messageReceived(FtpHandlerAdapter.java:62)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1015)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.ftpserver.listener.nio.FtpLoggingFilter.messageReceived(FtpLoggingFilter.java:85)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
        at org.apache.mina.filter.logging.MdcInjectionFilter.filter(MdcInjectionFilter.java:162)
        at org.apache.mina.filter.util.CommonEventFilter.messageReceived(CommonEventFilter.java:84)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:418)
        at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:257)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1128)
        at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:106)
        at org.apache.mina.core.session.IoEvent.run(IoEvent.java:89)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:763)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:755)
        at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:695)
        at java.base/java.lang.Thread.run(Thread.java:829)

[elecharny]

Ok, this is clearly not a problem in MINA FTP server code. At some point, when the RETR command is received, the following method is called:

        at com.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand.execute(RetrCommand.java:74)

then it tries to open a connection to a remote peer which fails (the top of the stack trace).

I suggest you ask the third party (tuya.com) about what's going on there.

[elecharny]

Some question:

• the fact the socket times out drives me to think it's more like a configuration issue on your side than a problem in the code. Can you check that?
• Which product are you using? Is it a tuya.com API based lib?

[sanshengshui]

I am an employee of tuya.com and am trying to set up an ftp service using FTP-server

I'll keep trying to figure out what caused it

Whether wget enters epsv mode and curl defaults to pasv mode

[elecharny]

Ok, so can you provide the code associated with om.tuya.sftp.protocol.gateway.service.protocol.command.RetrCommand?

[sanshengshui]

I made a preliminary determination that curl went to epsv mode by default.
The curl command is as follows:

➜  ~ curl --ftp-pasv -u 02982C3LaV:TjTv7Swb -O ftp://f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 198.19.10.106:8183...
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0< 220 Service ready for new user.
> USER 02982C3LaV
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0< 331 User name okay, need password for 02982C3LaV.
> PASS TjTv7Swb
< 230 User logged in, proceed.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Entering Passive Mode (|||13017|)
*   Trying 198.19.10.106:13017...
* Connecting to 198.19.10.106 (198.19.10.106) port 13017
* Connected to f1.iot17.com (198.19.10.106) port 8183 (#0)
> TYPE I
< 200 Command TYPE okay.
> SIZE 1698304726c2bcfc8265c.tar.gz
< 213 16483570
> RETR 1698304726c2bcfc8265c.tar.gz
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0< 150 File status okay; about to open data connection.
* Maxdownload = -1
* Getting file with size: 16483570
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0{ [0 bytes data]
* transfer closed with 16483570 bytes remaining to read
  0 15.7M    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
* Closing connection 0
curl: (18) transfer closed with 16483570 bytes remaining to read

When I use wget, pasv mode works fine

➜  ~ wget ftp://02982C3LaV:TjTv7Swb@f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz                    
--2023-10-27 11:10:17--  ftp://02982C3LaV:*password*@f1.iot17.com:8183/1698304726c2bcfc8265c.tar.gz
           => “1698304726c2bcfc8265c.tar.gz”
正在解析主机 f1.iot17.com (f1.iot17.com)... 198.19.10.106
正在连接 f1.iot17.com (f1.iot17.com)|198.19.10.106|:8183... 已连接。
正在以 02982C3LaV 登录 ... 登录成功！
==> SYST ... 完成。   ==> PWD ... 完成。
==> TYPE I ... 完成。 ==> 不需要 CWD。
==> SIZE 1698304726c2bcfc8265c.tar.gz ... 16483570
==> PASV ... 完成。   ==> RETR 1698304726c2bcfc8265c.tar.gz ... 完成。
长度：16483570 (16M) (非正式数据)

1698304726c2bcfc8265c.tar.gz          100%[======================================================================>]  15.72M  2.24MB/s  用时 7.3s    

2023-10-27 11:10:25 (2.16 MB/s) - “1698304726c2bcfc8265c.tar.gz” 已保存 [16483570]

my ftp-server code as follows:

        FtpServerFactory serverFactory = new FtpServerFactory();
        CommandFactoryFactory commandFactoryFactory = new CommandFactoryFactory();
        commandFactoryFactory.addCommand("STOR", new StorCommand());
        commandFactoryFactory.addCommand("RETR", new RetrCommand());
        commandFactoryFactory.addCommand("USER", new UserCommand());
        commandFactoryFactory.addCommand("PASS", new PassCommand());
        commandFactoryFactory.addCommand("CWD", new CwdCommand());
        commandFactoryFactory.addCommand("TYPE", new TypeCommand());
        commandFactoryFactory.addCommand("SIZE", new SizeCommand());
        commandFactoryFactory.addCommand("PASV", new PasvCommand());
        CommandFactory commandFactory = commandFactoryFactory.createCommandFactory();
        serverFactory.setCommandFactory(commandFactory);
        ListenerFactory listenerFactory = new ListenerFactory();
        listenerFactory.setPort(ftpConfig.getFtpPort());
        DataConnectionConfigurationFactory dataConnectionConfFactory = new DataConnectionConfigurationFactory();
        dataConnectionConfFactory.setPassivePorts(ftpConfig.getPassivePorts());
        String externalIP = ExternalIPUtils.getExternalIP(ftpConfig.getPublicIpv4Domain());
        ......
       Listener listener = listenerFactory.createListener();
        serverFactory.addListener("default", listener);
        Map<String, Ftplet> ftpLets = new HashMap();
        ftpLets.put("ftpService", new MyFtpPlet());
        serverFactory.setFtplets(ftpLets);
        ......
         server = serverFactory.createServer();

my RetrCommand as follows:

@Slf4j
public class RetrCommand extends RETR {

    @Override
    public void execute(FtpIoSession session, FtpServerContext context, FtpRequest request) {
        try {
            // argument check
            String fileName = request.getArgument();
            if (fileName == null) {
                session.write(LocalizedDataTransferFtpReply.translate(session, request,
                        context, FtpReply.REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS, "RETR", null, null));
                return;
            }

            // 24-10-2007 - added check if PORT or PASV is issued, see
            // https://issues.apache.org/jira/browse/FTPSERVER-110
            //TODO move this block of code into the super class. Also, it makes
            //sense to have this as the first check before checking everything
            //else such as the file and its permissions.
            DataConnectionFactory connFactory = session.getDataConnection();
            if (connFactory instanceof IODataConnectionFactory) {
                InetAddress address = ((IODataConnectionFactory) connFactory).getInetAddress();
                log.info("======>address:{}", JsonUtils.toString(address));
                if (address == null) {
                    session.write(new DefaultFtpReply(FtpReply.REPLY_503_BAD_SEQUENCE_OF_COMMANDS, "PORT or PASV must be issued first"));
                    return;
                }
            }

            // get data connection
            session.write(LocalizedFtpReply.translate(session, request, context, FtpReply.REPLY_150_FILE_STATUS_OKAY, "RETR", null));

            // send file data to client
            boolean failure = false;
            InputStream is = null;

            FtpFile file = null;
            DataConnection dataConnection;
            try {
                dataConnection = session.getDataConnection().openConnection();
            } catch (Exception e) {
                log.debug("Exception getting the output data stream", e);
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "RETR", null, null));
                return;
            }
            long transSz = 0L;
            try {
                try {
                    User user = session.getUser();
                    fileName = fileName.replace("/", "");
                    String objectKey = user.getHomeDirectory();
                    String fileUrl = objectKey + fileName;
                    log.info("name:{},fileUrl:{}", user.getName(), fileUrl);
                    IFtpFileService ftpFileService = SpringHolder.getBean(IFtpFileService.class);
                    is = ftpFileService.openUrlInputStreamByCetus(user.getName(), fileUrl);
                } catch (Exception e) {
                    session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                            FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "RETR", null, null));
                    e.printStackTrace();
                    return;
                }

                // transfer data
                transSz = dataConnection.transferToClient(session.getFtpletSession(), is);
                // attempt to close the input stream so that errors in
                // closing it will return an error to the client (FTPSERVER-119)
                if (is != null) {
                    is.close();
                }

                log.info("File downloaded {}", fileName);

                // notify the statistics component
                ServerFtpStatistics ftpStat = (ServerFtpStatistics) context
                        .getFtpStatistics();
                if (ftpStat != null) {
                    ftpStat.setDownload(session, file, transSz);
                }

            } catch (SocketException ex) {
                log.debug("Socket exception during data transfer", ex);
                failure = true;
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED,
                        "RETR", fileName, null, transSz));
            } catch (IOException ex) {
                log.debug("IOException during data transfer", ex);
                failure = true;
                session.write(LocalizedDataTransferFtpReply.translate(
                        session, request, context, FtpReply.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN,
                        "RETR", fileName, file, transSz));
            } finally {
                // make sure we really close the input stream
                IoUtils.close(is);
            }

            // if data transfer ok - send transfer complete message
            if (!failure) {
                session.write(LocalizedDataTransferFtpReply.translate(session, request, context,
                        FtpReply.REPLY_226_CLOSING_DATA_CONNECTION, "RETR", fileName, file, transSz));

            }
        } finally {
            session.resetState();
            session.getDataConnection().closeDataConnection();
        }
    }
}

The key code is：
dataConnection = session.getDataConnection().openConnection();

[sanshengshui]

One way to cheat is: Through what configuration can the client only go to PASV mode
😄

[elecharny]

@sanshengshui the diff I see is that curl uses ESPV while wget uses PASV. Is that a IPV6 connection?

[sanshengshui]

IPV4 connection