Media Transfer Protocol and > 4GB file transfers
Large file transfers with Media Transfer Protocol (MTP) are very mysterious. The most common search results are about Android users having difficulties with transferring large files and while, this has nothing to do with the MTP specification itself and more to do with the
largely deficient Android MTP implementation, MTP typically bears the users' wrath. I believe that part of the stigma associated with MTP has to do with Microsoft being instrumental in its inception. There are some others, for example, the name could have been better! (A person, apparently a storage expert, was actually unaware that Media Transfer Protocol, while focused on media files can actually handle other file types as well!). As to why Android decided MTP to be the default file system is something we can get into in another post but for now, let's get back to the curious case of large files.
The MTP spec could use some clarity for the section on writing large files. The fundamental issue with writes being that the MTP object metadata uses a 32 bit unsigned integer for the file size which limits the file size. The spec mentions that for a file size > 4GB, the size should be set to 0xFFFFFFFF. However, it doesn't clearly specify how to manage the transfer itself (when/how to stop!).
So, can you transfer a > 4GB file with MTP ? Yes, you can and there are two ways to implement it. The first is to use a ObjectPropList, specifically, support SendObjectPropList. This operation has a 8 byte field for object size. However, ObjectPropLists may not be supported by a lot of initiators and even if they are, initiators may decide not to use them for write support.
The second method is mostly reliant on the transport mechanism that MTP mostly relies on, USB. It probably even makes the file size field in the object metadata an optional item. USB guarantees that the end of the data phase will be marked by what is called a short packet. A short packet is a data packet whose size is less than wMaxPacketSize specified by the endpoint descriptor, basically the size of the data buffer in the endpoint. The snippet to handle the write operation in the responder could be something like:
total_size += incoming_datasize;
if (incoming_datasize < wMaxPacketSize) {
/* This is the last packet in this data phase */
write_buffer_to_file();
free_buffer();
}
There's a small hiccup here though. Typically, USB controllers will try to fill up their buffer before sending the data out. So, the above “if condition” may become true even before the actual end of the data phase and mark an incorrect end of data phase. Fixing this is easy though and relies on the fact that the incoming data size will be a multiple of 64 unless it's the end of the data phase:
total_size += incoming_datasize;
if ((incoming_datasize % 64) != 0) {
/* This is the last packet in this data phase */
write_buffer_to_file();
free_buffer();
}