Create ML Trouble Loading CSV to Train Word Tagger With Commas in Training Data

In the (hopefully) short-term, I am able to export the Numbers spreadsheet as a TSV and have written up a crude converter to generate JSON from it that Create ML can properly handle. However, that adds an extra step that I would hope could be eliminated by directly exporting from Numbers for use in Create ML.

Can you share the CSV file as exported by Numbers? All commas within a cell need to be quoted or escaped. In your example there are only a few quoted or escaped commas.

I have attached the CSV. The lines in question are 290 and 291. Create ML will interpret those as 288 and 289, since it's zero-based and excludes the header row. Importing will work with no problems up to that point and will work with no problems if I delete both rows before exporting to CSV. If you open the CSV in Numbers, everything looks fine.

The CSV file is correct, but I'm having a hard time understanding how the tokens are encoded. The word tagger needs an array of strings. So you need to go from that representation to an array of strings.

For example this line "?,.,!,"","","",:,;", after CSV processing becomes ?,.,!,",",",:,;. I assume it represents ["?", ".", "!", <comma>, <double quote>, ":", ";"], but you'll need to write custom code to make that interpretation. My suggestion is to use a different escape character to make that translation easier, for instance "?,.,!,\,,\"",:,;" and then interpret \ as an escape.

The other option is to use JSON encoding: "[""?"", ""."", ""!"", "","", ""\"""", "":"", "";""]" and then use this to decode:

var dataFrame = try DataFrame(
    contentsOfCSVFile: url,
    types: ["TOKENS": .data]
)
try dataFrame.decode([String].self, inColumn: "TOKENS", using: JSONDecoder())

Here is a more recent case to show what I'm trying to do, as that example with the punctuation was a proof-of-concept for testing. This includes a few commas within the text to be trained. Other examples include quotation marks.

Hodeeyáádą́ą́ʼ,Diyin,God,yótʼááh,hiníláii,índa,nahasdzáán,áyiilaa,.,Nahasdzáán,tʼáadoo,ánoolniní,da,",",índa,tʼáadoo,bikááʼ,siláhí,da,;,bikáaʼgi,tʼáá,átʼéé,nítʼééʼ,chahałheełgo,Diyin,God,biNíłchʼi,Diyinii,tó,yikááʼgóó,nahazleʼ,.,Áádóó,Diyin,God,ádííniid,",",Adinídíin,leʼ,.,Tʼáá,áko,adinídíín,hazlį́į́ʼ,.,Áko,Diyin,God,éí,adinídínígíí,yinééłʼį́įʼgo,bił,yáʼíítʼééh,",",áádóó,adinídínígíí,chahałheeł,yił,ałtsʼáyíínil,.

Adv,Adj,NSub,NObjPos,VPerf,Conj,NObj,VPerf,Punct,NSub,AdvNeg,VProg,PartNeg,Punct,Conj,AdvNeg,Adp,VImpf,PartNeg,Punct,Adp,Adv,VImpf,Adv,Adv,Adj,NSub,NSubPos,NSubPos,NAdp,AdpPos,VImpf,Punct,Conj,Adj,NSub,VPerf,Punct,NObj,VImp,Punct,Adv,Adv,NSub,VPerf,Punct,Adv,Adj,NSub,Pro,NObj,VPerfAdv,ProAdp,VPerf,Punct,Conj,NSub,NAdp,Adp,VPerf,Punct

The core problem is that Create ML does not seem to support several CSV escaping formats that various spreadsheet tools do (including Apple's own Numbers). Additionally, it does not support other file formats directly exported from Numbers that I could find. That makes commas and quotation marks difficult to include in any training data.

I've been able to get around this by writing my own tool that imports TSV files from Numbers and converts them to JSON files that Create ML accepts, adding 2 more steps to the training process each time. However, this post was originally about how to get Create ML to directly accept a Numbers CSV file without added steps every time. If it is not a bug, and Create ML just lacks the functionality, I will continue as I have with my custom work-around, and we can consider this issue resolved.