FYI, I append JSON files millions of times per file. Here's the code:
func saveBAToJSON(BA: myCustomClass) {
let fileName = "created from BA"
let ArchiveURL = getArchiveURL(fileName: fileName, ext: "JSON")
var ArchiveDirectoryPath = fileDirPath
let localFileManager = FileManager()
do {
let resources = try? ArchiveURL.resourceValues(forKeys: [.fileSizeKey])
let fileSize = resources?.fileSize ?? 0
/* The JSON file is appended many times and it must be limited in size otherwise memory is over taxed. maxFileSize is a global var that is machine specific. */
if fileSize > maxFileSize {
let dirContents = try! localFileManager.contentsOfDirectory(atPath: ArchiveDirectoryPath)
var elemFiles = 0
/* If a file would exceed MaxFileSize, start a new file copying it and appending an incremented counter to the name. */
for elem in 0..<dirContents.count { if dirContents[elem].contains(fileName) { elemFiles += 1 } }
try FileManager.default.copyItem(at: ArchiveURL, to: getArchiveURL(fileName: fileName+"-\(elemFiles)", ext: "JSON"))
try FileManager.default.removeItem(at: ArchiveURL)
}
} catch { print("\(Date.now):\(#file):\(#line) saveBAToJSON():fileSize", error) }
do {
let myJSONencoder = JSONEncoder()
myJSONencoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "+Infinity", negativeInfinity: "-Infinity", nan: "NaN")
var data = try myJSONencoder.encode(BA)
if let fileHandle = try? FileHandle(forWritingTo: ArchiveURL) {
defer {
fileHandle.closeFile()
}
/* This is the main trick: truncate the repeating JSON components from the top of file to be appended, strip the JSON components at the end of the file to append-to (after converting to base64) replacing it with a comma (in my case). Write to file. */
let truncateCount = UInt64(36 + filename.count + String(BA.myCustomClassElement1).count)
/* you must count chars in your specific JSON's to get this right. I'm sure this could be coded to adapt to any JSON file. */
let strippedRange = data.range(of: Data(base64Encoded: "my base64 encoded repeating components at end-of-file" )!)
let commaData = Data(base64Encoded: "LA==")!
data.replaceSubrange(strippedRange!, with: commaData )
let end = try fileHandle.seekToEnd()
if end > truncateCount {
try fileHandle.truncate(atOffset: end - truncateCount )
try fileHandle.write(contentsOf: data )
}
}
else { try data.write(to: ArchiveURL, options: [.atomic, .noFileProtection]) }
} catch { print("\(Date.now):\(#file):\(#line) saveBAToJSON():fileSize", error) }
}
Share and Enjoy
homage Eskimo
Armed with Scott's clue, ".localizedStandard", here's what works:
after collecting [URL] with "contentsOfDirectory(at:", parse-out a [String] from the URL components.
apply .sorted(using: .localizedStandard) and use in Picker()
from Picker() use the return value and re-assemble a path to use with "URL(fileWithPath:"
finish-off with JSONDecoder()
Gives a nice Finder-style Picker list and opens the file.
Thanks Scott
Ah, thank you! Yes, using .sortedKeys returned the desired behavior. It is curious that this must have been a default as evidenced by consistent formatting for over a year, that is, until now.
Thanks again.
Verified by commenting-out "DispatchQueue.main.async{ self.loop_count += 1 }"
A simple solution was to move the DQ between loop17 and loop16, the ProgressView works and no memory race.
Thanks Eskimo.
Sorry, my error. I had engineered my DispatchQueue's over two years ago. Re-engineering them uses all cores!
Found it...
“Build Phases”:”Compile Sources”:LAPACK functions.swift”. double-click and add: -DACCELERATE_NEW_LAPACK
hit return.
I had to re-launch XCode to get the alerts to stop.
Post not yet marked as solved
Is this work entirely CPU bound?
No GPU's involved, so I guess so. The app does nothing outside of the machine upon which it runs. Once the database is generated the app just becomes a viewing instrument.
This reply is two weeks late because for two weeks the machine was busy generating useless data. lol. (Down from a 300 year estimated compute time, so yay, progress!)
We can probably close this thread because I have no need to generate data on unknown machines. Thanks.
Post not yet marked as solved
My original intent was flawed, i. e., trying to optimize code using pre-compiler tests would only be useful for those who would be compiling, and not app users.
What do you plan to do with that info?
My app has nested for-loops sometimes eighteen deep. By strategically placing DispatchQueue's between the outer loops I can keep all my cpu's busy, minimizing compute time. In an app I can Switch() on the ncpu to the appropriate nest for optimization. As one who compiles the code, it's actually easier for me to edit the nests to my machines ncpu.
A second flaw in my thought process is that once the data is generated no-one need generate it again. Nevertheless, I love solving mysteries or maybe just pursuing solutions.
Post not yet marked as solved
Re-discovered eskimo's "//developer.apple.com/forums/thread/690310". Result:
launch(tool: URL(fileURLWithPath: "/usr/sbin/sysctl"), arguments: ["-n", "hw.ncpu" ], completionHandler: { (status, outputData) in
let output = String(data: outputData, encoding: .utf8) ?? ""
self.ncpu = (output as NSString).integerValue
print("done, status: \(status), output: \(self.ncpu)")
})
where I had declared ncpu elsewhere in the app.
Thank you eskimo!
Post not yet marked as solved
I have attempted to use sysctl() to retrieve the count. It prints the correct value to standard out, i.e., the debugger area but Idk how to capture it in swiftui. ".standardOutput" throws something called an "NSConcreteTask", and not the Int() I was expecting. NSConcreteTask has no entry in the "Developer Documentation" of Xcode, so I'm stuck.
let getCPUct = Process()
getCPUct.executableURL = URL(fileURLWithPath: "/usr/sbin/sysctl")
getCPUct.arguments = ["-n", "hw.ncpu" ]
do {
try getCPUct.run() // throws 6 to standard out
if let output = getCPUct.standardOutput as? Int {
print("#file:#line cpu count = \(output)")
return output
}
}
catch let error as NSError { print("Failed to execute sysctl", error) }
return 0
}
I have migrated my app to SwiftUI and this issue is solved in my post "EnvironmentObject as progressValue in ProgressView()"
Thanks
As for my apps "nested for-loops" I am pursuing DispatchQueue with multiple cpu's. See my post: "precompiler test for cpu count".
My six cpu Mini now requires three months of computations, not a thousand years.
Found a solution! Search: "quick-start/swiftui/how-to-show-progress-on-a-task-using-progress"
Use a Timer to allow ProgressView to update from EnvironmentObject vars.
Then you must config your app-file with the proper .environmentObject()
app-file:
import SwiftUI
@main
struct test_progressView_EnvObjApp: App {
var body: some Scene {
WindowGroup {
ContentView().environmentObject(Model.init(progressValue: 0.0))
}
}
}
Model-file:
import Foundation
import SwiftUI
class Model: ObservableObject {
@Published var t_step : Double
@Published var progressValue : Double
init (progressValue: Double){
self.t_step = 0.0
self.progressValue = 0.0
}
func stepThroughTime() {
var when = DispatchTime.now()
progressValue = 0.0
let _ = Timer.scheduledTimer(withTimeInterval: 2, repeats: true) { timer in
DispatchQueue.global().asyncAfter(deadline: when ) { [self] in
DispatchQueue.main.async { [self] in
if progressValue >= 0.9 { progressValue = 0.0 }
else { progressValue = progressValue + 0.1 }
}
when = when + .seconds(2)
}
}
}
}
ContentView-file:
import SwiftUI
struct ContentView: View {
@EnvironmentObject var model: Model
@State var progressValue = 0.0
let timer = Timer.publish(every: 2.0, on: .main, in: .common).autoconnect() // seconds
var body: some View {
VStack {
Button(
action: {
model.stepThroughTime()
}
)
{ Text("Search") }.environmentObject(model)
ProgressView(value: progressValue ).onReceive(timer) { _ in
progressValue = model.progressValue
}.padding()
.environmentObject(model)
}.environmentObject(model)
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
Thanks to Claude31 for the DispatchQueue-ProgressValue portion. See my post on the subject.
I looked into ndjson. It required a library download and had a number of conditions for which it would fail.
It turns out that editing and appending to my JSON file was not hard at all, due to the consistency of the JSON encoder.
Since eskimo provided the invaluable info that keyed archives don't support appending, I'll credit him.
Thank you.
eskimo, thanks for your reply.
Is there no way to to append-to-file piecemeal and then read it again?
If being keyed is the roadblock, I really don't need it as my original app stored as JSON text anyway. So, how can I save without being keyed? My understanding was that Apple would soon require "SecureCoding" which in turn required keying.
func loadFile(_ N: Int) -> classTwo? {
/* https://developer.apple.com/forums/thread/115643 OOPer https://www.hackingwithswift.com/read/12/3/fixing-project-10-nscoding */
let fileURL = URL(fileURLWithPath: "~/\" ).appendingPathExtension("gut")
let defaults = UserDefaults.standard
if let data = defaults.object(forKey: "classTwo") as? Data {
if FileManager.default.fileExists(atPath: fileURL.path) {
do {
if let BA = try? NSKeyedUnarchiver.unarchivedObject(ofClass: classTwo.self, from: data) {
return BA
}
}
}
}
return nil
}
