-
-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SFTP file read very slow, length is ignored #68
Comments
Hey @deze333 , sorry to say but the packet size limit you're running into is not a bug in Citadel nor SwiftNIO. The variable you linked is not used by Citadel or NIOSSH, but is a separate variable that references the same constant. It's maybe not a coincidence that both use the same value, as it's a more commonly used value for limiting message sizes in network protocols. The issue you're running into is caused by your SFTP server, likely openssh, which specifies a 1 mebibyte limit for any packet. The read limit is a result of that, and cannot be increased by a client such as Citadel. That being said, I think what we can do is look at parallelising this work - if the SFTP protocol allows for that. |
Thank you @Joannis for casting light on this problem. Looks like it's the Now I imagine it would be a non-trivial work to extend Citadel to support download parallelism. That would make a huge sense though. Fingers crossed! |
I'll look into it after my vacation. |
Even though
SFTPFile
methodsread(from offset:length:)
andreadAll()
request specificlength
of bytes to be read, the underlying NIO channel ignores that and seems to discard all data past its default chunk size value which is effectively:This results in very slow download times as the same portions of file are being multiple times but NIO stripping (or so it seems) all data past 131072 bytes for each request. Effectively the file is being read many times over again and again.
The only way to somewhat remedy the problem is to use the
length
of 131072 but the download speed is still too slow due to many requests needed with 128k chunks.The file's attributes are known and its
attributes.size
contains correct number of bytes. ThereadAll()
attempts to read the file at once using itsattributes.size
value.Code to reproduce:
Client:
Server:
Additional context
The log of reading a 5Mb file where
readAll()
is requesting full file download butlength
is being ignored and replaced by default NIO value.The text was updated successfully, but these errors were encountered: