Skip to content

Commit

Permalink
Added how to obtain the profile document corresponding to the query
Browse files Browse the repository at this point in the history
  • Loading branch information
@xyfsjq
xyfsjq committed Feb 28, 2024
1 parent 9103c14 commit 6a77571
Show file tree
Hide file tree
Showing 2 changed files with 371 additions and 0 deletions.
185 changes: 185 additions & 0 deletions docs/en/docs/admin-manual/http-actions/fe/get-profile-action.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
---
{
"title": "How to obtain the corresponding query profile",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# How to obtain the corresponding query profile

## Background

We often encounter situations where the execution time of the corresponding SQL is less than expected. In order to optimize the SQL to achieve the expected query delay, through the profile we can see what optimizations can be done. Now we will explain how to get the profile corresponding to the query in different environments.
## Request

`http://fe_ip:http_port`
`GET /api/profile`


## The Doris cluster can normally access the external network

1. Enable profile reporting parameters enable_profile

This parameter turns on the session variable. It is not recommended to turn this variable on globally.

```
--open variable
mysql> set enable_profile=true;
Query OK, 0 rows affected (0.00 sec)
--Confirm whether the variable is opened normally
mysql> show variables like '%profile%';
+----------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+----------------+-------+---------------+---------+
| enable_profile | true | false | 1 |
+----------------+-------+---------------+---------+
1 row in set (0.00 sec)
```

2. Execute the corresponding query

In the case of a cluster with multiple fes, you need to execute the corresponding query on the fes that enable profile reporting parameters. The parameters do not take effect globally.

```
--open variable
mysql> set enable_profile=true;
Query OK, 0 rows affected (0.00 sec)
--Confirm whether the variable is opened normally
mysql> show variables like '%profile%';
+----------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+----------------+-------+---------------+---------+
| enable_profile | true | false | 1 |
+----------------+-------+---------------+---------+
1 row in set (0.00 sec)
--Execute the corresponding query
mysql> select id,name from test.test where name like "%RuO%";
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| id | name |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| 1ZWXYGbb8nr5Pi29J4cEMyEMb | ZN1nqzBRSl1rTrr99rnX1aplxhRuOUTLw6so7rzjlRQ317gTPxh0dHljmrARDJjH7FjRkJW9c7YuUBmWikq7eNgmFKJPreWirDrGrFzUYH4eP6kDtSA3UTnNIIj |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
```

3. Get profile

When the cluster has multiple fes, you need to access the QueryProfile page of the fe http interface (http://fe_ip:http_port) that executes the corresponding query. Click the corresponding Profile ID to view the corresponding profile. You can also download the corresponding profile in the profile interface.

## Doris cluster’s access to the external network is restricted

When the cluster cannot access the external network normally, it needs to obtain the corresponding profile through the API (http://fe_ip:http_port/api/profile?query_id=). The ip and port refer to the ip and port corresponding to fe that executes the corresponding query. At this time, the first two steps of obtaining the profile corresponding to the query are the same as when accessing the external network normally. There will be a difference in the third step of obtaining the profile.

**Get profile**

- Find the corresponding Profile ID

```
--Find the Profile ID according to the corresponding query
mysql> show query profile "/";
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
| Profile ID | Task Type | Start Time | End Time | Total | Task State | User | Default Db | Sql Statement |
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
| 1b0bb22689734d30-bbe56e17c2ff21dc | QUERY | 2024-02-28 11:00:17 | 2024-02-28 11:00:17 | 7ms | EOF | root | | select id,name from test.test where name like "%RuO%" |
| 202fb174510c4772-965289e8f7f0cf10 | QUERY | 2024-02-25 19:39:20 | 2024-02-25 19:39:20 | 19ms | EOF | root | | select id,name from test.test where name like "%KJ%" |
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
2 rows in set (0.00 sec)
```

- Query profile and redirect profile to a text

```
template：curl -X GET -u user:password http://fe_ip:http_port/api/profile?query_id=1b0bb22689734d30-bbe56e17c2ff21dc > test.profile
[user@VM-10-6-centos profile]$ curl -X GET -u root:root http://127.0.0.1:8030/api/profile?query_id=1b0bb22689734d30-bbe56e17c2ff21dc > test.profile
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1211 0 1211 0 0 168k 0 --:--:-- --:--:-- --:--:-- 168k
```

- The returned profile line break is \ \n, which is inconvenient to analyze. You can replace \ \n with \n in a text editing tool.

```
[user@VM-10-6-centos profile]$ cat test.profile
{"msg":"success","code":0,"data":{"profile":"Query:\n Summary:\n
- Profile ID: 1b0bb22689734d30-bbe56e17c2ff21dc\n - Task Type: QUERY\n
- Start Time: 2024-02-28 11:00:17\n - End Time: 2024-02-28 11:00:17\n
- Total: 7ms\n - Task State: EOF\n - User: root\n - Default Db: \n
- Sql Statement: select id,name from test.test where name like \"%RuO%\"\n Execution Summary:\n
- Workload Group: \n - Analysis Time: 1ms\n
- Plan Time: 2ms\n - JoinReorder Time: N/A\n
- CreateSingleNode Time: N/A\n - QueryDistributed Time: N/A\n
- Init Scan Node Time: N/A\n - Finalize Scan Node Time: N/A\n
- Get Splits Time: N/A\n - Get PARTITIONS Time: N/A\n
- Get PARTITION FILES Time: N/A\n - Create Scan Range Time: N/A\n
- Schedule Time: N/A\n - Fetch Result Time: 0ms\n - Write Result Time: 0ms\n
- Wait and Fetch Result Time: N/A\n - Doris Version: doris-2.0.4-rc06-003a815b63\n
- Is Nereids: Yes\n - Is Pipeline: Yes\n - Is Cached: Yes\n
- Total Instances Num: 0\n - Instances Num Per BE: \n
- Parallel Fragment Exec Instance Num: 48\n - Trace ID: \n"},"count":0}
```
- The effect after replacement is as follows

```
{"msg":"success","code":0,"data":{"profile":"Query:
Summary:
- Profile ID: 1b0bb22689734d30-bbe56e17c2ff21dc
- Task Type: QUERY
- Start Time: 2024-02-28 11:00:17
- End Time: 2024-02-28 11:00:17
- Total: 7ms
- Task State: EOF
- User: root
- Default Db:
- Sql Statement: select id,name from test.test where name like \"%RuO%\"
Execution Summary:
- Workload Group:
- Analysis Time: 1ms
- Plan Time: 2ms
- JoinReorder Time: N/A
- CreateSingleNode Time: N/A
- QueryDistributed Time: N/A
- Init Scan Node Time: N/A
- Finalize Scan Node Time: N/A
- Get Splits Time: N/A
- Get PARTITIONS Time: N/A
- Get PARTITION FILES Time: N/A
- Create Scan Range Time: N/A
- Schedule Time: N/A
- Fetch Result Time: 0ms
- Write Result Time: 0ms
- Wait and Fetch Result Time: N/A
- Doris Version: doris-2.0.4-rc06-003a815b63
- Is Nereids: Yes
- Is Pipeline: Yes
- Is Cached: Yes
- Total Instances Num: 0
- Instances Num Per BE:
- Parallel Fragment Exec Instance Num: 48
- Trace ID:
"},"count":0}
```


186 changes: 186 additions & 0 deletions docs/zh-CN/docs/admin-manual/http-actions/fe/get-profile-action.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
---
{
"title": "如何获取对应 query 的 profile",
"language": "zh-CN"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# 如何获取对应 query 的 profile

## 背景

我们时常遇到对应 sql 执行时间不及预期的情况，为了优化 sql 达到预期查询时延，通过 profile 我们能够看出可以做哪些优化。现在说明在不同环境下应该如何拿到对应 query 的 profile。

## 请求

`http://fe_ip:http_port`
`GET /api/profile`


## Doris 集群能够正常访问外网

1. 开启 profile 上报参数 enable_profile

该参数开启的是 session 变量，此变量不建议全局开启。

```
--开启变量
mysql> set enable_profile=true;
Query OK, 0 rows affected (0.00 sec)
--确认变量是否正常开启
mysql> show variables like '%profile%';
+----------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+----------------+-------+---------------+---------+
| enable_profile | true | false | 1 |
+----------------+-------+---------------+---------+
1 row in set (0.00 sec)
```

2. 执行对应 query

集群在多个 fe 的情况下，需要到开启 profile 上报参数的 fe 上执行对应 query, 参数并没有全局生效。

```
--开启变量
mysql> set enable_profile=true;
Query OK, 0 rows affected (0.00 sec)
--确认变量是否正常开启
mysql> show variables like '%profile%';
+----------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+----------------+-------+---------------+---------+
| enable_profile | true | false | 1 |
+----------------+-------+---------------+---------+
1 row in set (0.00 sec)
--执行对应 query
mysql> select id,name from test.test where name like "%RuO%";
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| id | name |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| 1ZWXYGbb8nr5Pi29J4cEMyEMb | ZN1nqzBRSl1rTrr99rnX1aplxhRuOUTLw6so7rzjlRQ317gTPxh0dHljmrARDJjH7FjRkJW9c7YuUBmWikq7eNgmFKJPreWirDrGrFzUYH4eP6kDtSA3UTnNIIj |
+---------------------------+-----------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
```

3. 获取 profile

集群在多个 fe 的情况下，需要访问执行对应 query 的 fe http 界面( http://fe_ip:http_port )的 QueryProfile 页面,点击对应 Profile ID 查看对应 profile，还可以在 profile 界面下载对应 profile

## Doris 集群访问外网受到限制

集群不能正常访问外网时，需要通过 api 的方式拿到对应 profile (http://fe_ip:http_port/api/profile?query_id=),ip 和端口是指执行对应 query 的 fe 对应 ip 和端口。此时获取对应 query 的 profile 步骤前两步和正常访问外网时是一样的,第三步获取 profile 时会有差别

**获取 porfile**

- 找到对应 query id

```
--根据对应 query 找到 Profile ID
mysql> show query profile "/";
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
| Profile ID | Task Type | Start Time | End Time | Total | Task State | User | Default Db | Sql Statement |
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
| 1b0bb22689734d30-bbe56e17c2ff21dc | QUERY | 2024-02-28 11:00:17 | 2024-02-28 11:00:17 | 7ms | EOF | root | | select id,name from test.test where name like "%RuO%" |
| 202fb174510c4772-965289e8f7f0cf10 | QUERY | 2024-02-25 19:39:20 | 2024-02-25 19:39:20 | 19ms | EOF | root | | select id,name from test.test where name like "%KJ%" |
+-----------------------------------+-----------+---------------------+---------------------+-------+------------+------+------------+-------------------------------------------------------+
2 rows in set (0.00 sec)
```

- 查询 profile 并将 profile 重定向到一个文本中

```
模板：curl -X GET -u user:password http://fe_ip:http_port/api/profile?query_id=1b0bb22689734d30-bbe56e17c2ff21dc > test.profile
[user@VM-10-6-centos profile]$ curl -X GET -u root:root http://127.0.0.1:8030/api/profile?query_id=1b0bb22689734d30-bbe56e17c2ff21dc > test.profile
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1211 0 1211 0 0 168k 0 --:--:-- --:--:-- --:--:-- 168k
```

- 返回的 profile 换行符为 \ \n 分析起来很不方便, 可以在文本编辑工具中将 \ \n 替换为 \n

```
[user@VM-10-6-centos profile]$ cat test.profile
{"msg":"success","code":0,"data":{"profile":"Query:\n Summary:\n
- Profile ID: 1b0bb22689734d30-bbe56e17c2ff21dc\n - Task Type: QUERY\n
- Start Time: 2024-02-28 11:00:17\n - End Time: 2024-02-28 11:00:17\n
- Total: 7ms\n - Task State: EOF\n - User: root\n - Default Db: \n
- Sql Statement: select id,name from test.test where name like \"%RuO%\"\n Execution Summary:\n
- Workload Group: \n - Analysis Time: 1ms\n
- Plan Time: 2ms\n - JoinReorder Time: N/A\n
- CreateSingleNode Time: N/A\n - QueryDistributed Time: N/A\n
- Init Scan Node Time: N/A\n - Finalize Scan Node Time: N/A\n
- Get Splits Time: N/A\n - Get PARTITIONS Time: N/A\n
- Get PARTITION FILES Time: N/A\n - Create Scan Range Time: N/A\n
- Schedule Time: N/A\n - Fetch Result Time: 0ms\n - Write Result Time: 0ms\n
- Wait and Fetch Result Time: N/A\n - Doris Version: doris-2.0.4-rc06-003a815b63\n
- Is Nereids: Yes\n - Is Pipeline: Yes\n - Is Cached: Yes\n
- Total Instances Num: 0\n - Instances Num Per BE: \n
- Parallel Fragment Exec Instance Num: 48\n - Trace ID: \n"},"count":0}
```
- 替换后的效果如下

```
{"msg":"success","code":0,"data":{"profile":"Query:
Summary:
- Profile ID: 1b0bb22689734d30-bbe56e17c2ff21dc
- Task Type: QUERY
- Start Time: 2024-02-28 11:00:17
- End Time: 2024-02-28 11:00:17
- Total: 7ms
- Task State: EOF
- User: root
- Default Db:
- Sql Statement: select id,name from test.test where name like \"%RuO%\"
Execution Summary:
- Workload Group:
- Analysis Time: 1ms
- Plan Time: 2ms
- JoinReorder Time: N/A
- CreateSingleNode Time: N/A
- QueryDistributed Time: N/A
- Init Scan Node Time: N/A
- Finalize Scan Node Time: N/A
- Get Splits Time: N/A
- Get PARTITIONS Time: N/A
- Get PARTITION FILES Time: N/A
- Create Scan Range Time: N/A
- Schedule Time: N/A
- Fetch Result Time: 0ms
- Write Result Time: 0ms
- Wait and Fetch Result Time: N/A
- Doris Version: doris-2.0.4-rc06-003a815b63
- Is Nereids: Yes
- Is Pipeline: Yes
- Is Cached: Yes
- Total Instances Num: 0
- Instances Num Per BE:
- Parallel Fragment Exec Instance Num: 48
- Trace ID:
"},"count":0}
```


0 comments on commit 6a77571

Please sign in to comment.