iOS 10.3 (APFS) iTunes file sharing problem with Asian characters

I've got multiple reports from users about problem after updating to iOS 10.3.

Users can't access files that are transferred using iTunes file sharing after upgrading to iOS 10.3.

This problem doesn't happen when transferring files with only alpha numeric characters.

Bug was present when transferring file names with Korean and Japanese.

It transfers without any problem and shows up correctly in iTunes(file sharing).


But the file is not accessible from the app.

The file is listed in NSFileManager::contentsOfDirectoryAtPath.

But when checking the file with NSFileManager::attributesOfItemAtPath

it fails with

Error Domain=NSCocoaErrorDomain Code=260 "The file “KoreanJapaneseCharacters.txt” couldn’t be opened because there is no such file." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/0353E581-F595-4E83-83C0-AAFBF4589A8A/Documents/KoreanJapaneseCharacters.txt, NSUnderlyingError=0x17025d880 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

( Had to replace filename because this website doesn't seem to like foreign characters. )


The bug is not reproducible on my devices, but I got the error message using a TestFlight build from a bug reporter.


And files transferred before the iOS 10.3 update (with foreign characters) doesn't have problem accessing. And transferring english file names and then renaming to Korean file names inside the app works without problem.


Anyone got similar bug reports? I would love to fix the problem, but not sure what I can do to fix it.

I think it has something do with the APFS update.

Replies

APFS and HFS Plus differ in the way they handle normalisation in file system names; APFS is normalisation preserving while HFS Plus always normalises file names to a (relatively odd) normal form. The Apple File System Guide has recently been updated to discuss this change. However, if you get the contents of a directory and then pass exactly those names to a routine that accesses the files, that should always work.

It’s going to be hard to debug the problem being seen by your customers without being able to reproduce it. If you’re working with a user who can reproduce the problem, you could send them a debug build that dumps out the problematic names as a UTF-8 hex dump. You should be able to then add test code to your app to produce a file with that name, and from there debug your import process.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Apple developers (or anyone) can reproduce the problem at anytime if they are getting Korean (or Asian?) name files to Pages, Numbers, and Keynotes from Windows iTunes File Sharing. (macOS is OK)


Pages, Numbers, and Keynores in iOS 10.3~10.3.1 can not get Korean name files via Korean Windows iTunes File Sharing ever.


(NOTE) If you are using macOS, the problem does not exist. Most developers use macOS so that they may not see the problem... Please use (Korean? or Asian?) Windows to check this problem.


At this time, all I can do is praying Apple developers fix the problem as soon as possible but they don't... It is a very serious bug... I reported it to Bug Reporter yesterday but I can not expect they are going to read mine in a near future... because they are looked like always doing something better and more important than...

1. You can reproduce the problem if you are using Windows iTunes File Sharing. If not, please try Korean or Japanese Windows if possible. Any files with Korean name will not be accessible... (I had to spend 4 days to see the problem because I am using macOS also for developing. Finally I'd tested Windows iTunes and got the problem.)


2. Unfortunately I've found that only Apple can solve the problem. You may find a way to access the file using system calls lke open(3) , etc. but current FileManager will always fail to access the file. No copyItem, no moveItem, no removeItem, etc.

I was able to reproduce the bug by transferring from Korean Windows 10 using iTunes.

Thanks to musebook for the information about reproducing the bug.


I've created txt files from Windows, Mac and transferred using Windows iTunes and Mac iTunes. I also created a file directly from iOS.


Here's the array returned from [[NSFileManager defaultManager] contentsOfDirectoryAtPath:path error:&error];


"Mac \U1112\U1161\U11ab\U1100\U1173\U11af.txt",

"Windows \Ud55c\Uae00.txt",

"iOS \U1112\U1161\U11ab\U1100\U1173\U11af.txt"

All files have same "'Hangul" Korean 2 letters with prefix where they came from. Mac and iOS share the same Unicode and file from Windows have different Unicode. When I NSLog each NSString from the array, I could see correct Korean output. I also see all files from iTunes file sharing correctly both from Windows and Mac.

When doing attributesOfItemAtPath with:

NSString *filepath = [path stringByAppendingPathComponent:filename];

NSDictionary *dict=[fileManager attributesOfItemAtPath:filepath error:&error];

The "Windows <HanGul>.txt" fails with error:

Error Domain=NSCocoaErrorDomain Code=260 "The file “Windows <HanGul>.txt” couldn’t be opened because there is no such file." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/9EB241A6-D14F-4DF0-B7D2-42F0FBDF89DC/Documents/Windows <HanGul>.txt, NSUnderlyingError=0x17024b8e0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

I also tried using [filename decomposedStringWithCanonicalMapping]. When I convert Windows filename, it becomes same Unicode as Mac/iOS.

But it also does not return attributesOfItemAtPath.

I also tried reading the file with [NSData dataWithContentsOfFile:]

When I use fopen and fread the file can be read! (without the decomposedStringWithCanonicalMapping).

So I will have to drop to POSIX/C APIs to read the Korean/Japanese files transferred from Windows iTunes.

Here's the code that I used to test:

            NSArray* dirs = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:path
                                                                                error:NULL];
            [dirs enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
                NSString *filename = (NSString *)obj;
                NSString *extension = [[filename pathExtension] lowercaseString];
                if ([extension isEqualToString:@"txt"]) {
                    // filename = [filename decomposedStringWithCanonicalMapping];
                    NSLog(@"filename = %@", @[filename]);
                    NSString *filepath = [path stringByAppendingPathComponent:filename];
                    NSLog(@"filepath = %@", @[filepath]);
                    NSDictionary *dict=[fileManager attributesOfItemAtPath:filepath error:NULL];
                    NSLog(@"dict = %@", dict);
                    NSData *data = [NSData dataWithContentsOfFile:filepath];
                    NSLog(@"len(data) = %lu", (unsigned long) [data length]);
                
                    FILE *pFile = fopen([filepath UTF8String], "r");
                    char mystring [100];
                    if (pFile == NULL) {
                        NSLog(@"Error opening file");
                    } else {
                        fgets (mystring , 100 , pFile);
                        NSLog(@"data = %s", mystring);
                        fclose (pFile);
                    }
                }
            }];

Output for "Windows <HanGul>.txt"

filename = (
    "Windows \Ud55c\Uae00.txt"
)
filepath = (
    "/var/mobile/Containers/Data/Application/956ED4A3-94F2-4EA2-A14D-C86F0DF3A693/Documents/Windows \Ud55c\Uae00.txt"
)
dict = (null)
len(data) = 0
data = text

Two different Unicodes for a Korean filename are...


One is String.decomposedStringWithCanonicalMapping (macOS and iOS)

And the other is String.precomposedStringWithCanonicalMapping (Windows)


I've already tested to change the filename from Windows to decomposed but there was no success to make it accessible... It's just for your information not to spend your time like mine.

First off, please, files bugs on this and post the number here. There are a number of different problems in this area and it's important that these issues get documented.


You can access these files with NSFileManager and Cocoa, but you need to use specific APIs and test things carefully. Two things are going on here:


-NSString implicitly normalized pathes whenever it interacts with them, which is what's causing the core behavior you're seeing. NSURL does NOT share this behavior and can be used. All of the NSURL NSFileManager APIs should work fine, none of the path/NSString APIs should be used.


-Seperately, some APIs that take NSURL arguments actually end up extracting the path and then using NSString. So, for example, while dataWithContentsOfFile doesn't work for the reason above, dataWithContentsOfURL also fails because it's not really using NSURL under the hood. On the other handle, NSFileHandle fileHandleForReadingFromURL (and variants) all work fine.


All of these issues are a mix of intended (though unexpected) behaviors and bugs, which is why it's so important that bugs get filed on whatever you find.


-Kevin

My app uses AVFoundation and passes path using [NSURL fileURLWithPath:].

I wasn't able to read files from AVFoundation. It seems NSURL automatically converts path to decomposedStringWithCanonicalMapping.


So I used this workaround before accessing files: (rename files using POSIX API!)


    NSArray* dirs = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:path
                                                                        error:NULL];
    [dirs enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
        NSString *filename = (NSString *)obj;
      
        NSString *dfilename = [filename decomposedStringWithCanonicalMapping];
      
        if (![filename isEqualToString:dfilename]) {
            NSString *filepath = [path stringByAppendingPathComponent:filename];
            NSString *dfilepath = [path stringByAppendingPathComponent:dfilename];
            rename([filepath UTF8String], [dfilepath UTF8String]);
        }
    }];


I don't think this will be applicable to all applications, but it makes my app usable NOW without too much code change.

Hope Apple fixes this soon.

"All of the NSURL NSFileManager APIs should work fine"


FileManager.default.copyItem(at: srcURL, to: dstURL) throws an error the file does not exist

FileManager.default.moveItem(at: srcURL, to: dstURL) throws an error the file does not exist

FileManager.default.removeItem(at: url) throws an error the file does not exist


URL and FileManager always Failed to access Korean name files added from Windows iTunes File Sharing.


=> All of the NSURL NSFileManager APIs does not work at all in this case.

I reported to Bug Reporter a couple of days ago and the number is 31435344

Yes, I'm afraid your correct. I had tested NSFileHandle (which works fine) and several of the high level APIs (which did not), but I had not fully tested NSFileManager and was basing the statement about NSURL on what I had been told.

This error is reproduced for files files added from Windows iTunes File Sharing on Russian Windows 10 and have russian letters in filename.

Guys from iMazing have fixed the issue: https://imazing.com/blog/ios-10-3-broke-itunes-file-sharing-on-windows

This bug is fixed in iOS 11 beta3!

This bug is fixed in iOS 11 beta3!

To be clear, there’s both a long-term and a short-term strategy for addressing this issue. WWDC 2017 Session 715 What’s new in Apple File System has the details.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"