Hello All, New member here so hopefully my post adheres to Dev Forum standards. My Platform team and I are attempting to gather a bunch of our customer/service data via an automated process where we code a script to obtain data from the App Store Connect dashboard of our app. We need to web scrape most likely with the App-Store-Scraper library. This reason for this is the following challenge we have run into previously while utilizing the App Store Connect API: When we have attempted to fetch data we have seen that the attributes we are looking for cannot be obtained with the query parameters available. Certain reports can only be obtained for DAILY query param values; certain information, such as "File Size" from the "State" report, and "revenue net" of Apple's commissions on sales from the "Proceeds" report within the App Store Connect dashboard, are not able to be obtained via the API, to my knowledge. Of course it can be obtained manually by downloading it through the dashboard, but we want to automate this process. My question is with regards to web scraping: Will we be able to obtain all the information we need by web scraping the "Event, State, Proceeds" reports that one can find by navigating from the left-hand-side tabs within the App Store Connect dashboard? This information contains PII, but since we have rights to our customer data we should be able to maintain confidentiality by following industry standards. Please let me know if anyone has any experience with this kind of task before and I look forward to hearing from you.

Hello fellow developers! I am interested in obtaining the following information regarding one of our Home Screen widgets: Pull the following data about the "my_app_widget" homepage widget in total and by month for the last 12 months (Sep '23 - Sep '24): Widget install events Unique widget installs Device type (iPad, iOS). Thus far, I am using AWS Lambda and have the proper credentials within a Secrets Manager and I keep getting a 409 error from this code: const AWS = require('aws-sdk'); const axios = require('axios'); const jose = require('jose'); const crypto = require('crypto'); const s3 = new AWS.S3(); const secretsManager = new AWS.SecretsManager(); const cloudwatchlogs = new AWS.CloudWatchLogs(); const logGroupName = '/aws/lambda/my_lambda_function'; const logStreamName = '2024/11/05/[$LATEST]XXXXXXXXXXXXXX'; const getSecret = async (secretName) => { const secret = await secretsManager.getSecretValue({ SecretId: secretName }).promise(); return JSON.parse(secret.SecretString); }; const logError = async (message) => { const params = { logGroupName, logStreamName, logEvents: [ { message: JSON.stringify(message), timestamp: Date.now() } ] }; try { await cloudwatchlogs.putLogEvents(params).promise(); } catch (error) { console.error('Failed to log to CloudWatch:', error); } }; const getJwtToken = async (keyId, issuerId, privateKeyPem) => { try { const privateKey = crypto.createPrivateKey({ key: privateKeyPem, format: 'pem', type: 'pkcs8' }); const payload = { iss: issuerId, exp: Math.floor(Date.now() / 1000) + 20 * 60, aud: 'appstoreconnect-v1' }; const token = await new jose.SignJWT(payload) .setProtectedHeader({ alg: 'ES256', kid: keyId }) .sign(privateKey); return token; } catch (error) { await logError({ error: 'Error generating JWT', details: error }); throw new Error('JWT generation failed'); } }; const fetchAndUploadReport = async (url, token, s3Key, isAnalytics = false, body = null) => { try { const headers = { 'Authorization': `Bearer ${token}`, 'Accept': isAnalytics ? 'application/json' : 'application/a-gzip', 'Content-Type': 'application/json' }; const response = await axios({ method: isAnalytics ? 'post' : 'get', url: url, headers: headers, data: body, responseType: isAnalytics ? 'json' : 'stream' }); if (response.status === 200) { let bodyContent = isAnalytics ? JSON.stringify(response.data) : response.data; await s3.upload({ Bucket: 'my_bucket_name', Key: s3Key, Body: bodyContent }).promise(); return { statusCode: 200, body: `${s3Key} fetched and uploaded successfully` }; } else { await logError({ error: 'API response error', status: response.status, statusText: response.statusText, url }); return { statusCode: response.status, body: response.statusText }; } } catch (error) { await logError({ error: 'Error fetching report', url, details: error }); return { statusCode: 500, body: JSON.stringify(`Error fetching ${s3Key}`) }; } }; exports.handler = async (event) => { const secretName = 'AppStoreConnectPrivateKey'; try { const secretData = await getSecret(secretName); const { keyId, issuerId, private_key: privateKeyPem } = secretData; const token = await getJwtToken(keyId, issuerId, privateKeyPem); const startDate = '2023-09-01'; const endDate = '2024-09-30'; const apiEndpoints = [ { url: `https://api.appstoreconnect.apple.com/v1/analyticsReportRequests`, // Changed to request report s3Key: 'my_folder/my_subfolder/unique_widget_installs.json', isAnalytics: true, body: { "data": { "type": "analyticsReportRequests", "attributes": { "accessType": "ONGOING", "name": "Home Screen Widget Installs", "category": "APP_USAGE" }, "relationships": { "app": { "data": { "type": "apps", "id": "YYYYYYYYYYYY" } } } } } } ]; Returns the following error : { "error": "Error fetching report", "url": "https://api.appstoreconnect.apple.com/v1/analyticsReportRequests", "details": { "message": "Request failed with status code 409", "name": "AxiosError", "stack": "AxiosError: Request failed with status code 409\n at settle (/var/task/node_modules/axios/dist/node/axios.cjs:2019:12)\n at IncomingMessage.handleStreamEnd (/var/task/node_modules/axios/dist/node/axios.cjs:3135:11)\n at IncomingMessage.emit (node:events:531:35)\n at IncomingMessage.emit (node:domain:488:12)\n at endReadableNT (node:internal/streams/readable:1696:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n at Axios.request (/var/task/node_modules/axios/dist/node/axios.cjs:4287:41)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async fetchAndUploadReport (/var/task/index.js:68:26)\n at async Promise.all (index 0)\n at async exports.handler (/var/task/index.js:174:25)", "config": { "transitional": { "silentJSONParsing": true, "forcedJSONParsing": true, "clarifyTimeoutError": false }, "adapter": [ "xhr", "http", "fetch" ], "transformRequest": [ null ], "transformResponse": [ null ], "timeout": 0, "xsrfCookieName": "XSRF-TOKEN", "xsrfHeaderName": "X-XSRF-TOKEN", "maxContentLength": -1, "maxBodyLength": -1, "env": {}, So I am not sure if the JSON body I am passing this URL endpoint are correct or if this is the correct API to be using to attempt to obtain this information. Please let me know when you have the chance and I look forward to hearing from you.