Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding functionality to push metrics and diagnostics to AppInsights using Open Telemetry #1700

Draft
wants to merge 87 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
244a4b9
cosmos SDK and client upgrades (Moving Naveen's work to Azure YCSB fork)
RaviTella Jul 24, 2022
d5ef9ba
Update azurecosmos.properties
RaviTella Jul 24, 2022
11507aa
CODE_OF_CONDUCT.md committed
microsoftopensource Jul 25, 2022
6849cc7
LICENSE committed
microsoftopensource Jul 25, 2022
ed2b8f7
README.md updated to template
microsoftopensource Jul 25, 2022
a0d3719
SECURITY.md committed
microsoftopensource Jul 25, 2022
9a2d92b
SUPPORT.md committed
microsoftopensource Jul 25, 2022
9cf3923
Merge pull request #2 from RaviTella/user/ratella/cosmos-update
RaviTella Jul 26, 2022
8a46ed6
log4j2 setup
RaviTella Jul 31, 2022
4cfdf49
Update pom.xml
RaviTella Jul 31, 2022
d59c864
Merge branch 'Azure:main' into user/ratella/logging
RaviTella Jul 31, 2022
e73375a
logging
RaviTella Jul 31, 2022
4d8eef3
Merge branch 'user/ratella/logging' of https://github.com/RaviTella/Y…
RaviTella Jul 31, 2022
487d90e
Logging marker
RaviTella Jul 31, 2022
449184b
Update AzureCosmosClient.java
RaviTella Jul 31, 2022
5434dae
Update log4j2.xml
RaviTella Jul 31, 2022
53391ab
logging
RaviTella Jul 31, 2022
7ff4175
Update log4j2.xml
RaviTella Jul 31, 2022
0279b43
Update log4j2.xml
RaviTella Jul 31, 2022
0e28ab5
Update AzureCosmosClient.java
RaviTella Jul 31, 2022
d2a71e2
Update log4j2.xml
RaviTella Aug 2, 2022
6623744
Update log4j2.xml
RaviTella Aug 2, 2022
b25f0fd
Update log4j2.xml
RaviTella Aug 2, 2022
2512c7e
Update log4j2.xml
RaviTella Aug 2, 2022
821f984
Update log4j2.xml
RaviTella Aug 2, 2022
71ad4ed
separating diagnostic logs
RaviTella Aug 4, 2022
7d95703
Adding long number support for long run
simplynaveen20 Aug 4, 2022
60b6826
adding multiple markers
RaviTella Aug 5, 2022
d4e206a
Update log4j2.xml
RaviTella Aug 5, 2022
99ffe0c
Update log4j2.xml
RaviTella Aug 5, 2022
f5d2724
Update log4j2.xml
RaviTella Aug 5, 2022
aac96a9
Update log4j2.xml
RaviTella Aug 5, 2022
408cbd7
Update log4j2.xml
RaviTella Aug 5, 2022
24a053e
Update log4j2.xml
RaviTella Aug 5, 2022
2cde15f
Update log4j2.xml
RaviTella Aug 5, 2022
d230587
Update log4j2.xml
RaviTella Aug 5, 2022
828f0f6
changing marker name
RaviTella Aug 6, 2022
d07f11c
Update log4j2.xml
RaviTella Aug 6, 2022
ab8a5ae
Update log4j2.xml
RaviTella Aug 6, 2022
3c01e91
Update log4j2.xml
RaviTella Aug 6, 2022
45e2b26
query and patch markets added
RaviTella Aug 6, 2022
d1200d5
Enabling request diagnostics for query
RaviTella Aug 6, 2022
89e9e8f
Merge pull request #5 from Azure/users/nakumars/addingLongNumberSuppo…
RaviTella Aug 6, 2022
0f16588
UUID as ID for insert
RaviTella Aug 6, 2022
cb4ff20
Update AzureCosmosClient.java
RaviTella Aug 6, 2022
721e5b6
Update AzureCosmosClient.java
RaviTella Aug 7, 2022
227bea0
Update AzureCosmosClient.java
RaviTella Aug 7, 2022
dee7122
Upgrade to Log4J2 and logging for request diagnostics
RaviTella Aug 8, 2022
3fcf68c
Updating cosmos sdk to latest 4.34.0
simplynaveen20 Aug 9, 2022
747f84d
Merge pull request #7 from Azure/users/nakumars/updateCosmosSDKTo4.34.0
RaviTella Aug 9, 2022
0fc84ac
Merge branch 'master' into users/ratella/requestdiagnostics
RaviTella Aug 9, 2022
2398fd8
Revert "Merge pull request #7 from Azure/users/nakumars/updateCosmosS…
RaviTella Aug 9, 2022
f56210a
Cosmos sdk version update to 4.34.0
RaviTella Aug 9, 2022
41f4d08
Merge pull request #8 from RaviTella/users/ratella/requestdiagnostics
RaviTella Aug 9, 2022
91c71f9
Merge pull request #9 from Azure/request_diagnostics
simplynaveen20 Aug 11, 2022
743806b
Adding AppInsight monitering support
simplynaveen20 Aug 25, 2022
e182773
Merge pull request #10 from Azure/users/nakumars/addingLiveMetricsRep…
RaviTella Aug 31, 2022
2582c93
directing Cosmos operation exception to separate file
RaviTella Sep 16, 2022
436e65d
check style fix
RaviTella Sep 16, 2022
e18b31d
Merge pull request #11 from Azure/users/ratella/exception_logging
RaviTella Sep 16, 2022
5053973
Fixing threading issue on micrometer integration on cosmos ycsm
simplynaveen20 Oct 20, 2022
d42531b
Merge pull request #12 from Azure/users/nakumars/fixingThreadingIssue…
RaviTella Oct 21, 2022
c7d042e
Update README.md
RaviTella Oct 31, 2022
2da5753
Update README.md
RaviTella Oct 31, 2022
5fd86e3
Merge pull request #13 from Azure/users/ratella/readme
RaviTella Oct 31, 2022
34fbb53
bumping cosmos version sdk to 4.39.0
simplynaveen20 Jan 10, 2023
1c10969
fixing comment in cosmos client file
simplynaveen20 Jan 10, 2023
0c6efe4
Merge pull request #14 from Azure/users/nakumars/updatingCosmosSDKto4-39
simplynaveen20 Jan 10, 2023
24092d4
Adding cosmos in log diagnotics and exception log folder
simplynaveen20 Jan 12, 2023
c2cb207
Merge pull request #15 from Azure/users/nakumars/updatingCosmosLogLoc…
simplynaveen20 Jan 12, 2023
c33973b
Bumping cosmos sdk version (#17)
simplynaveen20 Feb 21, 2023
ff11219
Creating new reactive stream Mongo driver (#18)
simplynaveen20 Mar 7, 2023
a4a8822
Adding preferred region list config to Azure Cosmos client (#21)
simplynaveen20 May 9, 2023
873a278
fix for log configuration for delete workload (#22)
RaviTella Jun 15, 2023
cea1da2
Add GH Action Workflow
karianna Jul 15, 2023
899efed
Merge pull request #24 from karianna/add_gh_action
simplynaveen20 Jul 28, 2023
60613e9
Cosmos SDK upgraded to 4.52.0 (#36)
RaviTella Dec 7, 2023
6c9d4f1
Changed micrometer implementation to send info to Open Telemetry
Mar 18, 2024
0e30cc5
Replaced * with single imports
Mar 18, 2024
a1ecd2c
parameterized cosmos client diagnostics log options
Mar 19, 2024
49def82
updated ycsb added jvm args
Mar 20, 2024
f55a279
removed default args
Mar 20, 2024
0e3e853
added setenv.sh file for JAVA_OPTS
Mar 20, 2024
7c824e5
updated pom file
Mar 21, 2024
52ce2e3
removed the setenv.sh
Mar 21, 2024
95c9b41
added details in readme for sending telemetry to appInsights
Mar 22, 2024
91d52cd
Addressed PR comment
Mar 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/pr-builder.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: YCSB build with Maven Wrapper

on:
pull_request:
branches: [ main ]
paths-ignore:
- '**.md'

jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
java: [11, 17]

steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3

- name: Set up JDK ${{ matrix.java }}
uses: actions/setup-java@5ffc13f4174014e2d4d4572b3d74c3fa61aeb2c2 # 3.11.0
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: 'maven'

- name: Build with Maven Wrapper
run: ./mvnw clean install site
9 changes: 9 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Microsoft Open Source Code of Conduct

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).

Resources:

- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) Microsoft Corporation.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
96 changes: 25 additions & 71 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,80 +1,34 @@
<!--
Copyright (c) 2010 Yahoo! Inc., 2012 - 2016 YCSB contributors.
All rights reserved.
# Project

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You
may obtain a copy of the License at
This repository is a fork of the original YCSB repository. We had to create this fork as there was no response to our pull requests from the YCSB maintainers.

http://www.apache.org/licenses/LICENSE-2.0
This fork contains the following updates to Cosmos DB binding:
- Cosmos DB Java SDK 4.28.0
- Upgrade to Log4J 2
- Client diagnostic and exception logging to separate files
- Micrometer integration

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License. See accompanying
LICENSE file.
-->
Additionally, the “operationcount” type in YCSB core, has been changed to long from int to allow for large “operationcount” value.

YCSB
====================================
[![Build Status](https://travis-ci.org/brianfrankcooper/YCSB.png?branch=master)](https://travis-ci.org/brianfrankcooper/YCSB)
We will keep this fork up to date with the upstream YCSB repository and push changes from here, back to the upstream YCSB repository if it becomes active again.
## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

Links
-----
* To get here, use https://ycsb.site
* [Our project docs](https://github.com/brianfrankcooper/YCSB/wiki)
* [The original announcement from Yahoo!](https://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/)
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

Getting Started
---------------
## Trademarks

1. Download the [latest release of YCSB](https://github.com/brianfrankcooper/YCSB/releases/latest):

```sh
curl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz
tar xfvz ycsb-0.17.0.tar.gz
cd ycsb-0.17.0
```

2. Set up a database to benchmark. There is a README file under each binding
directory.

3. Run YCSB command.

On Linux:
```sh
bin/ycsb.sh load basic -P workloads/workloada
bin/ycsb.sh run basic -P workloads/workloada
```

On Windows:
```bat
bin/ycsb.bat load basic -P workloads\workloada
bin/ycsb.bat run basic -P workloads\workloada
```

Running the `ycsb` command without any argument will print the usage.

See https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload
for a detailed documentation on how to run a workload.

See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties for
the list of available workload properties.


Building from source
--------------------

YCSB requires the use of Maven 3; if you use Maven 2, you may see [errors
such as these](https://github.com/brianfrankcooper/YCSB/issues/406).

To build the full distribution, with all database bindings:

mvn clean package

To build a single database binding:

mvn -pl site.ycsb:mongodb-binding -am clean package
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
41 changes: 41 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->

## Security

Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).

If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.

## Reporting Security Issues

**Please do not report security vulnerabilities through public GitHub issues.**

Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).

If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).

You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc).

Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:

* Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
* Full paths of source file(s) related to the manifestation of the issue
* The location of the affected source code (tag/branch/commit or direct URL)
* Any special configuration required to reproduce the issue
* Step-by-step instructions to reproduce the issue
* Proof-of-concept or exploit code (if possible)
* Impact of the issue, including how an attacker might exploit the issue

This information will help us triage your report more quickly.

If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.

## Preferred Languages

We prefer all communications to be in English.

## Policy

Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).

<!-- END MICROSOFT SECURITY.MD BLOCK -->
25 changes: 25 additions & 0 deletions SUPPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# TODO: The maintainer of this repo has not yet edited this file

**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?

- **No CSS support:** Fill out this template with information about how to file issues and get help.
- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.

*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*

# Support

## How to file issues and get help

This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.

For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE
FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
CHANNEL. WHERE WILL YOU HELP PEOPLE?**.

## Microsoft Support Policy

Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
2 changes: 1 addition & 1 deletion azurecosmos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,4 +125,4 @@ following location:
### 3. FAQs

### 4. Example command
./bin/ycsb run azurecosmos -P workloads/workloadc -p azurecosmos.primaryKey=<your key eg:45fgt...==> -p azurecosmos.uri=https://<your account>.documents.azure.com:443/ -p recordcount=100 -p operationcount=100
./bin/ycsb run azurecosmos -P workloads/workloadc -p azurecosmos.primaryKey=<your key eg:45fgt...==> -p azurecosmos.uri=https://<your account>.documents.azure.com:443/ -p recordcount=100 -p operationcount=100
25 changes: 22 additions & 3 deletions azurecosmos/conf/azurecosmos.properties
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
# See https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips-java-sdk-v4-sql for details on some of the options below.

# Azure Cosmos DB host uri (ex: https://p3rf.documents.azure.com:443/) and primary key.
# azurecosmos.primaryKey =
# azurecosmos.uri =
# azurecosmos.primaryKey =
# azurecosmos.uri =

# Database to be used, if not specified 'ycsb' will be used.
# azurecosmos.databaseName = ycsb
Expand All @@ -31,6 +31,10 @@
# The default is false to reduce output size.
# azurecosmos.includeExceptionStackInLog = false

# Determines if full request diagnostics need to be printed for high latency requests.
# The default is -1(no diagnostics at all)to reduce output size.
# azurecosmos.diagnosticsLatencyThresholdInMS = -1

# The value to be appended to the user-agent header.
# In most cases, you should leave this as "azurecosmos-ycsb".
# azurecosmos.userAgent = azurecosmos-ycsb
Expand All @@ -54,7 +58,7 @@
# Set the maximum retry duration in seconds.
# azurecosmos.maxRetryWaitTimeInSeconds = 30

# Set the value of the connection pool size in gateway mode.
# Set the value of the connection pool size in gateway mode.
# azurecosmos.gatewayMaxConnectionPoolSize = 30

# Set the value of the max connections per endpoint in direct mode.
Expand Down Expand Up @@ -82,3 +86,18 @@
# Sets the preferred page size when scanning.
# Default value is -1.
# azurecosmos.preferredPageSize = -1

# output file location
# exportfile =

# application insight connection String
# azurecosmos.appInsightConnectionString =

# Cosmos Client Diagnostics logs options
# azurecosmos.pointOperationLatencyThresholdInMS =
# azurecosmos.nonPointOperationLatencyThresholdInMS =
# azurecosmos.requestChargeThreshold =

# Sets the comma separated preferred region list.
# Default value is null.
# azurecosmos.preferredRegionList =
45 changes: 28 additions & 17 deletions azurecosmos/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,27 +34,38 @@ LICENSE file.
<groupId>com.azure</groupId>
<artifactId>azure-cosmos</artifactId>
<version>${azurecosmos.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.18.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.18.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.18.0</version>
</dependency>
<dependency>
<groupId>site.ycsb</groupId>
<artifactId>core</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-azure-monitor</artifactId>
<version>1.9.2</version> <!-- {x-version-update;io.micrometer:micrometer-registry-azure-monitor;external_dependency} -->
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>applicationinsights-agent</artifactId>
<version>3.5.1</version>
</dependency>
</dependencies>
</project>
Loading