log retention on iOS sysdiagnose

Question

Hello All,

We get sysdiagnose files from iOS devices all the time and I started to see some inconsistencies in the data. In some cases the system_logs.logarchive can go back 30 days and others I only see 8 days.

When we convert the logs to a text file at times they can be 200MB and other times 1100MB

log show /system_logs.logarchive --info --debug > sysdiagnose_decrypted.log

More importantly, the application we create logs to this but some users can only see 24 hours back when the actual log goes back 12 days.

Example:

---first entry
2019-09-10 10:13:10.570199-0400 0xc022     Default     0x169e3              108    8    dasd: (DuetActivitySchedulerDaemon) [com.apple.duetactivityscheduler:bar] Recent Applications: 

2019-09-24 10:12:21.751882-0400 0x13588d   Default     0x0                  62     0    assertiond: [com.apple.assertiond:process_info] [MobileMail:4254] Setting jetsam priority to 0 [0x2000]

--First app entry
2019-09-24 11:17:33.859478-0400 0x137839   Default     0x58525a             62     0    assertiond: [com.apple.assertiond:process_info] [App tag :4629] Adding client: 

2019-09-24 11:17:35.421143-0400 0x13784e   Default     0x0                  4629   0    App tag : [INFO][2019-09-24 15:17:35 +0000] - Device is locked data is unavailable

Questions:

1. Does anyone know how the log retention works with sysdiagnose?

2. Why would we see our app at times only log back 24h and other times ~8 or 30 days?

3. Does device restarts alter the application data retention?

4. Apple has extra debug profiles to enhance/enable debug logs. Does anyone know how they might affect application data retention or the entire log retention?

Files and Storage

3.4k

Posted by

rpayne

Reply

Add a Comment

Answer 1

Log retention is a complex balancing act between multiple factors, including:

The rate at which log messages are generated
The size of those messages
The nature of those messages, and specifically whether the log system configuration causes the messages to persist
Disk space
Flash wear

The specific algorithm is not documented, and my understanding is that it changes regularly. As such, I don’t have any answers for questions 1 through 3.

With regards question 4:

Apple has extra debug profiles to enhance/enable debug logs. Does anyone know how they might affect application data retention or the entire log retention?

there’s two parts to this:

What do these profiles do?
What affect do they have on log retention rates?

It’s easy to answer the first question. These profiles are CMS-signed property lists, so you can decode them with the

security

tool. For example, the mDNSResponder for iOS profile looks like this:

$ security cms -D -i mDNSResponder.mobileconfig 
…
<plist version="1.0">
<dict>
    <key>ConsentText</key>
    …
    <key>DurationUntilRemoval</key>
    <real>1209600</real>
    <key>PayloadContent</key>
    <array>
        <dict>
            …
            <key>Subsystems</key>
            <dict>
                <key>com.apple.mDNSResponder</key>
                <dict>
                    <key>DEFAULT-OPTIONS</key>
                    <dict>
                        <key>Enable-Oversize-Messages</key>
                        <true/>
                        <key>Enable-Private-Data</key>
                        <true/>
                        <key>Level</key>
                        <dict>
                            <key>Enable</key>
                            <string>Info</string>
                            <key>Persist</key>
                            <string>Info</string>
                        </dict>
                    </dict>
                </dict>
            </dict>
        </dict>
    </array>
    …
</dict>
</plist>

If you’re up to speed on the unified logging architecture, this will look very familiar. If not, a good place to start is the discussion of the

config

subcommand in the

log

man page.

With regards their affect on log retention, the example above shows that a profile can cause more messages to persist, which is one of the factors I listed earlier.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Posted by

eskimo

Add a Comment

Answer 2

Log retention is a complex balancing act between multiple factors, including:

The rate at which log messages are generated
The size of those messages
The nature of those messages, and specifically whether the log system configuration causes the messages to persist
Disk space
Flash wear

The specific algorithm is not documented, and my understanding is that it changes regularly. As such, I don’t have any answers for questions 1 through 3.

With regards question 4:

Apple has extra debug profiles to enhance/enable debug logs. Does anyone know how they might affect application data retention or the entire log retention?

there’s two parts to this:

What do these profiles do?
What affect do they have on log retention rates?

It’s easy to answer the first question. These profiles are CMS-signed property lists, so you can decode them with the

security

tool. For example, the mDNSResponder for iOS profile looks like this:

$ security cms -D -i mDNSResponder.mobileconfig 
…
<plist version="1.0">
<dict>
    <key>ConsentText</key>
    …
    <key>DurationUntilRemoval</key>
    <real>1209600</real>
    <key>PayloadContent</key>
    <array>
        <dict>
            …
            <key>Subsystems</key>
            <dict>
                <key>com.apple.mDNSResponder</key>
                <dict>
                    <key>DEFAULT-OPTIONS</key>
                    <dict>
                        <key>Enable-Oversize-Messages</key>
                        <true/>
                        <key>Enable-Private-Data</key>
                        <true/>
                        <key>Level</key>
                        <dict>
                            <key>Enable</key>
                            <string>Info</string>
                            <key>Persist</key>
                            <string>Info</string>
                        </dict>
                    </dict>
                </dict>
            </dict>
        </dict>
    </array>
    …
</dict>
</plist>

If you’re up to speed on the unified logging architecture, this will look very familiar. If not, a good place to start is the discussion of the

config

subcommand in the

log

man page.

With regards their affect on log retention, the example above shows that a profile can cause more messages to persist, which is one of the factors I listed earlier.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Posted by

eskimo

Add a Comment

Answer 3

Thanks for the detailed info.

Can the log tool config and or stream commands be ran on a connected iOS device

Posted by

rpayne

Add a Comment

Answer 4

Can the log tool
config
and or
stream
commands be ran on a connected iOS device

No. There’s a complex balance here between security, privacy, and functionality here (and especially developer-level functionality). If you have specific requirements in this space, please describe them in one or more enhancement requests.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Posted by

eskimo

Add a Comment

Answer 5

@rpayne quoting what you said "but some users can only see 24 hours back when the actual log goes back 12 days."

Q1-By this do you mean there is a way to see that 12 days log?Do you mean that for the same sysdiagnose file ,some users see more information than others?

Q2-If you can only see 24hr info then how do you know it collected data for 12 days?
please answer this
I need help with this thing

Posted by

evolve3806

Add a Comment

log retention on iOS sysdiagnose

Accepted Reply

Replies