PHP中循环使用curl调用接口导致内存溢出如何解决
1. 运行环境
lnmp
1). 当前使用的 Laravel 版本?
laravel 版本:5.4
2). 当前使用的 php版本?
PHP 版本:7.1
3). 当前系统 ubuntu
2. 问题描述?
在laravel中循环调用接口拉取抖音数据导致内存溢出,每次内存溢出都是在curl_exec函数中,问一下这种情况有什么好的解决方法吗?循环调用是不可避免的
private function curl($url, $postData)
{
$ch = curl_init($url);
$options = [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_POSTFIELDS => $postData
];
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
if (curl_errno($ch)) {
Log::info(curl_error($ch));
} else {
$data = json_decode($response, true);
}
curl_reset($ch);
curl_close($ch);
unset($ch);
return $data;
}
贴一下目前的代码
$pullDay = date('Y-m-d', strtotime('-3 day'));
$pullDay = '2023-10-26';
$pageSize = 10000; //单次查询数量 抖音平台支持最大10000
$url = 'https://developer.toutiao.com/api/apps/taskbox/query_task_video_daily_data/';
//从数据库获取查询的记录
$record = DistributionDyTaskPullRecord::query()
->where('billing_date', $pullDay)
->orderByDesc('id')
->first();
$lastRequestEndTime = strtotime($pullDay) + 86399; //视频发布结束参数的最大时间
if (!$record) {
$pageNo = 1;
$publishStartTime = strtotime(date('Y-m-d', strtotime('-81 day')));
$publishEndTime = strtotime('+1 week -1 second', $publishStartTime);
} else {
if ($record->is_need_page) {
$pageNo = $record->page + 1;
$publishStartTime = $record->publish_start_time;
$publishEndTime = $record->publish_end_time;
} else {
$pageNo = 1;
$publishStartTime = $record->publish_end_time + 1;
$publishEndTime = strtotime('+1 week -1 second', $publishStartTime);
}
if ($record->status == 504 || $record->error_code == 4001010) {
$pageNo = $record->page;
$publishStartTime = $record->publish_start_time;
$publishEndTime = $record->publish_end_time;
}
//当所有条件满足的时候 就不在继续拉取
if ($record->status == 1 && $record->error_code == 0
&& $record->publish_end_time >= $lastRequestEndTime
&& $record->is_need_page == 0) return;
}
if ($publishEndTime > $lastRequestEndTime) $publishEndTime = $lastRequestEndTime;
// DistributionDyTaskSubmissionVideoDetail::query()->where('date', $pullDay)->delete();
$disTask = DistributionTask::all()->pluck('payment_allocate_ratio', 'task_id');
$packSetting = PackSetting::query()->where('id', 34)->first();
$post_data = [
'access_token' => $packSetting->access_token,
'appid' => $packSetting->appid,
'page_no' => $pageNo,
'page_size' => $pageSize,
'video_publish_start_time' => $publishStartTime,
'video_publish_end_time' => $publishEndTime,
'billing_date' => $pullDay,
];
$insertRecord = [
'page' => $pageNo,
'publish_start_time' => $publishStartTime,
'publish_end_time' => $publishEndTime,
'billing_date' => $pullDay,
];
try {
$data = GuzzleHelper::post($url, [ 'query' => $post_data ]);
$insertRecord['error_code'] = $data['error'] ?? 0;
$insertRecord['data_count'] = count($data['results']);
$insertData = [];
if ($data['error'] == 0 && count($data['results']) > 0) {
foreach ($data['results'] as $val) {
$new = [
'video_id' => $val['VideoId'] ?? 0,
'task_id' => $val['TaskId'] ?? 0,
'task_name' => $val['TaskName'] ?? '',
'publish_time' => $val['PublishTime'] ?? 0,
'author' => $val['Author'] ?? '',
'douyin_id' => $val['DouyinID'] ?? '',
'video_title' => $val['VideoTitle'] ?? '',
'micro_app_title' => $val['MicroAppTitle'] ?? '',
'video_views' => $val['video_views'] ?? 0,
'likes' => $val['Likes'] ?? 0,
'comments' => $val['Comments'] ?? 0,
'shares' => $val['Shares'] ?? 0,
'video_link' => $val['VideoLink'] ?? '',
'expected_profit' => $val['ExpectedProfit'] ?? 0,
'gmv_1d' => $val['GMV1d'] ?? 0,
'refund_gmv_1d' => $val['RefundGMV1d'] ?? 0,
'billing_gmv_1d' => $val['BillingGMV1d'] ?? 0,
'billing_refund_gmv_1d' => $val['BillingRefundGMV1d'] ?? 0,
'ad_share_cost_1d' => $val['AdShareCost1d'] ?? 0,
'feed_ad_share_cost_1d' => $val['FeedAdShareCost1d'] ?? 0,
'active_cnt_1d' => $val['ActiveCnt1d'] ?? 0,
'date' => $val['date'] ?? '',
'client_name' => $val['ClientName'] ?? '',
'payment_allocate_ratio' => $disTask[$val['TaskId']] ?? 0,
];
$insertData[] = $new;
}
$batches = array_chunk($insertData, 1000);
foreach ($batches as $batch) {
DistributionDyTaskSubmissionVideoDetail::insert($batch);
}
}
if (count($data['results']) < $pageSize) {
$insertRecord['is_need_page'] = 0;
} else {
$insertRecord['is_need_page'] = 1;
}
} catch (\Exception $e) {
Log::info('抖音数据拉取异常,拉取参数为' . json_encode($post_data) . '。异常信息' . $e->getMessage());
if ($e->getCode() == 504) {
$insertRecord['status'] = $e->getCode();
}
}
$insertRecord['created_at'] = Carbon::now();
DistributionDyTaskPullRecord::insert($insertRecord);
}
之前分页是从1-N 的样子和$pullDay这个参数是递增的类似10-21,10-22这样循环
找到原因了兄弟们