Skip to content
This repository has been archived by the owner on Jul 16, 2022. It is now read-only.

Commit

Permalink
Zio2 rc4 (#400)
Browse files Browse the repository at this point in the history
* Consolidate Client JVM tto serve WebSocket Protocol and Prometheus metrics

* Enable ZIO Core metrics for example

* Migrating the Prometheus encoding to ZIO 2 RC3

* Migrate to ZIO 2.0 RC3 / ZIO-JSON

* More refactoring, Refactored Samples

* Working on Migration to ZIO2 RC3

* Cleaning up compile errors after migrating to ZIO-Json

* Work on Json serialization / deserialization based on ZIO Json

* Finalize Client message serialization

* Apply formatting

* Finalize Json (de)serialization and tests

* Finalize protocol serialization with ZIO Json

* Refactor JS API to recent changes in metrics API

* Refactoring ScalaJS client for new API

* Apply formatting

* Finalize JS client refactoring

* Fix test in Prometheus Encoder

* Adding ZIO Json based dashboard im/export

* Ensure the grid-rows and grid-cols CSS defs are included in the
tailwinf generated CSS

* Merge stashed changes.

* ZIO 2.0 RC3 -> RC4

* Adjust to changes in ZIO 2 RC4

* Fix selection for individual keys in Ui

* Work on Json serialization / deserialization based on ZIO Json

* Apply formatting

* Finalize JS client refactoring

* Fix test in Prometheus Encoder

* Adding ZIO Json based dashboard im/export

* Ensure the grid-rows and grid-cols CSS defs are included in the
tailwinf generated CSS

* Adjust to changes in ZIO 2 RC4

* Fix selection for individual keys in Ui

* Finalize refactoring of StatsdListener

* ReExport Grafana Dashboard after updating to RC4 - Update Dashboard
ScreenShot

* Working on StatsDListener config

* Adding skeleton for StatsdListener Spec

* Make sure to use a login shell

* [WIP] First pass at a New Relic encoder.

* Common code for creating a New Relic Summary extracted.  'NewRelicEncoders' -> 'NewRelicEncoder'.

* [WIP] Top level array of 'metric' objects is now properly encoded.
- Generators added for Counter, Gauge and Frequency,
- New set of JsonAssertions added to streamline testing.
- 'NewRelicEncoderSpec' added and is verifying Counter, Gauge and Frequency.

* [WIP] Updated build.sbt so as to be able to resolve snapshots.

* ZIO RC4 support for min and max added to NewRelicEncoder.

* Code formatting, name changes, test refactor and Histogram generator added.

* Formatting applied.

* More formatting.

* Added a new architectural apporach to the services in New Relic.  They might be generic enough to use elsewhere.

* WIP:  Additional testing.

* Fixed versions

* Switching to the new Polling Model for Backends

I also went ahead with a first pass at a standard agent API
backend.  Still needs work and vetting but it's a start.

* Fixed race condition that occurs when polling rate
exceeds batching rate.

* Moved snapshot acquisition outside the mutex to
reduce the chance of lost metrics.

* Back pressuring logic added with relase valve.

* Comments

* MetricPublisher will no longer fail just returns a Result type.

NewRelicEncoder Histogram and Summary no longer produce buckets.

* MetricAgent refactoring for performance

- Preformance tests added for MetricAgent.
- No longer using 'groupedWithin' instead there is now optional thottling and and a max publishing size.

* encoding and publshing have been separated so that
publishing batching is correct.

* Refactor StatsdListener to use Encoder

* just formatting

* Basic refactoring after merging Metric Agent work

* Moving files from zio core

* Pulling in MetricEvent update

* Bump versions to latest ZIO 2 SNAPSHOTS

* Just a PoC for a MetricHub we can brainstorm against.

* Refining the API towards MetricEvents / Publishers hooked up with the
generic polling client

* MetricEvent with diffing smart constructor.

* More PoCs around MetricEvent, moved old PoCs to the attic

* Added the alternate Sonatype snapshot repo.

* Made 'retrieveNext' safer.

* Clean up commented-out code.

* Using local snapshot for zio-http

* NewRelicPublisher and spec are now working :D

* Integration test added for New Relic Publisher

- Code clean up to reflect current direction of ZMX.
- License added where it was missing.

* scalafmt

* scaladmt on test. added command alias, 'formatAll'.

* scalafmt on build.sbt.

* Added a test aspect 'ignoreIf' to provide better feedback
and ignore the NewRelicPublisherTest if the NR API Key
env var is not present.

* Fixed broken test cases.

* scalafmt

* Refactoring the Prometheus Backend towards the new MetricEncoder and
MetricPublisher API

* Implemented full-featured New Relic server example.

It uses the pre-existing 'InstrumentedSample' program to create
sample data along with include both jvm and ZIO telemetry.

I added a bunch of convenience accessors and layers to make user
experience as painless as possible.

* Fix trait name.

* Gauges should always be sent.

* Added exported New Relic dashboard settings.

* Refactor Stats Encoder / Publisher to new API.
Refactor and fix tests.
Add a CollectingPublisher to collect encoded metrics in test cases

* Added temporary rate shaping to the NewRelicPublisher.

* Integrated newest changes from 'series/2.x' plus snapshot that fixes hung process.

* Updated to the most recent ZIO 2.0 snaphot

* Refactor to a proper ZMX App Hierarchy.

I did a bunch of refinement around ZMXApp and NewRelicApp.

ZMXApp now bootstraps both the MetricClient, internal JVM
tracking and internal ZIO tracking.  This tracking can be
controlled via the flags provided by 'InternalMetrics#Settings'.

Classes extending ZMXApp can provide additional environmental
dependencies and can hook into the 'environmentalLayer' by
override 'additionalEnvironment'

NewRelicApp extends from ZMXApp and uses 'additionalEnvironment'
to register the 'NewRelicListener' with the 'MetricClient.
It also bootstraps publisher's event stream queue.

All '#Settings' now have 'live' layers that can pull settings from
environment variables and provide sane defaults where appropriate.

Many of the utility methods that were in package objects have found
homes in logically associated companion objects.

* scalfmt

* Made 'getWithDefault' safe.

* Spelling

* ZLayer.succeed -> ZEnvironment

* Reduced alternative "bloat" and NewRelicPublisher more configurable.

* Fix broken test compilation.

* Updates for ZIO 2.0 RC6

* Remove some commented-out code, some small refactors and some renaming.

* Small tweaks.

* Fix failed compilation.

* Use values provided by setting in NR publisher, bump versions based on ZIO 2 RC 6

* Move ScalaJS client to ZIO Insight
Rename packages from zio.zmx to zio.metrics.connectors

* Cleanup module structure
Cleanup build file

* Refactor tests to new package structure

* Moving ScalaJS client doc to ZIO Insight

* Move MetricClient to internal connectors pkg.

* Just starting some comments

* Refactor Statsd backend to latest ZLayers for ZIO 2

* Clean up backend API to Layers

* Just formatting

* Finalise the new layer structure

* Clean up test code

* Bring back the sample application

* Just refactor the sample app a bit

* Adding all available backends to sampleApp

* Update dashboard examples

* correct socket handling for StatsD

* Finalise sample app testing, working for all backends

* Remove obsolete code from the attic

* Remove unnecessary Version references

Co-authored-by: Scott T Weaver <scott.t.weaver@gmail.com>
  • Loading branch information
atooni and scottweaver authored May 29, 2022
1 parent 3ea7287 commit 23aa33a
Show file tree
Hide file tree
Showing 100 changed files with 2,609 additions and 7,209 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -422,3 +422,6 @@ website/node_modules
website/build
website/i18n/en.json
website/static/api

# The grafana data directory
data
9 changes: 9 additions & 0 deletions .scalafix.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
rules = [
DisableSyntax
ExplicitResultTypes
LeakingImplicitClassVal
NoAutoTupling
NoValInForComprehension
ProcedureSyntax
RemoveUnused
]
40 changes: 31 additions & 9 deletions .scalafmt.conf
Original file line number Diff line number Diff line change
@@ -1,15 +1,37 @@
version = "2.7.5"
version = "3.2.1"

runner.dialect = "scala213source3"

maxColumn = 120
align = most
align.preset = most
continuationIndent.defnSite = 2
assumeStandardLibraryStripMargin = true
docstrings = JavaDoc
docstrings.style = Asterisk
docstrings.wrap = no
lineEndings = preserve
includeCurlyBraceInSelectChains = false
danglingParentheses = true
spaces {
inImportCurlyBraces = true
}
includeCurlyBraceInSelectChains = true
danglingParentheses.preset = true
optIn.annotationNewlines = true

rewrite.rules = [SortImports, RedundantBraces, SortModifiers]
align.arrowEnumeratorGenerator = true

project.excludePaths = [
"glob:**/metals.sbt"
]

rewrite.rules = [SortModifiers, RedundantBraces, RedundantParens, PreferCurlyFors, Imports]
rewrite.imports.sort = scalastyle
rewrite.imports.groups = [
[ "java\\..*"],
["scala\\..*"],
["com\\..*"],
["org\\..*"],
["zio\\..*"]
]
rewrite.redundantBraces.stringInterpolation = true
rewrite.trailingCommas.style = always

verticalMultiline.atDefnSite = true
verticalMultiline.arityThreshold = 4

newlines.penalizeSingleSelectMultiArgList = false
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@

# Summary

ZIO ZMX provides pluggable backends into existing Application Performance Monitoring (APM) solutions,
including backends for New Relic, StatsD, DataDog and Prometheus.

TODO: Review sections below

ZIO ZMX allows you to add diagnostics and metrics to any ZIO application.

ZIO ZMX features:
Expand Down
40 changes: 0 additions & 40 deletions benchmarks/src/main/scala/zio/zmx/TrackingFibersBenchmark.scala

This file was deleted.

26 changes: 0 additions & 26 deletions benchmarks/src/main/scala/zio/zmx/package.scala

This file was deleted.

147 changes: 36 additions & 111 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,31 +1,36 @@
import BuildHelper._

inThisBuild(
List(
organization := "dev.zio",
homepage := Some(url("https://zio.github.io/zio.zmx/")),
licenses := List("Apache-2.0" -> url("http://www.apache.org/licenses/LICENSE-2.0")),
developers := List(
organization := "dev.zio",
homepage := Some(url("https://zio.github.io/zio.zmx/")),
licenses := List("Apache-2.0" -> url("http://www.apache.org/licenses/LICENSE-2.0")),
developers := List(
Developer(
"jdegoes",
"John De Goes",
"john@degoes.net",
url("http://degoes.net")
url("http://degoes.net"),
),
Developer(
"softinio",
"Salar Rahmanian",
"code@softinio.com",
url("https://www.softinio.com")
)
url("https://www.softinio.com"),
),
),
pgpPassphrase := sys.env.get("PGP_PASSWORD").map(_.toArray),
pgpPublicRing := file("/tmp/public.asc"),
pgpSecretRing := file("/tmp/secret.asc"),
scmInfo := Some(
ScmInfo(url("https://github.com/zio/zio.zmx/"), "scm:git:git@github.com:zio/zio.zmx.git")
pgpPassphrase := sys.env.get("PGP_PASSWORD").map(_.toArray),
pgpPublicRing := file("/tmp/public.asc"),
pgpSecretRing := file("/tmp/secret.asc"),
resolvers ++= Seq(
"s01 Sonatype OSS Snapshots" at "https://s01.oss.sonatype.org/content/repositories/snapshots",
"Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
),
testFrameworks := Seq(new TestFramework("zio.test.sbt.ZTestFramework"))
)
scmInfo := Some(
ScmInfo(url("https://github.com/zio/zio.zmx/"), "scm:git:git@github.com:zio/zio.zmx.git"),
),
testFrameworks := Seq(new TestFramework("zio.test.sbt.ZTestFramework")),
),
)

addCommandAlias("fmt", "all scalafmtSbt scalafmt test:scalafmt")
Expand All @@ -35,119 +40,39 @@ lazy val commonSettings = Seq()

lazy val root =
(project in file("."))
.aggregate(coreJS, coreJVM, clientJS, clientJVM, examples)
.settings(
publish / skip := true
)
.enablePlugins(BuildInfoPlugin)

lazy val core =
crossProject(JSPlatform, JVMPlatform)
.in(file("core"))
.settings(
stdSettings("zio.zmx"),
libraryDependencies ++= Seq(
"dev.zio" %%% "zio" % Version.zio,
"dev.zio" %%% "zio-streams" % Version.zio,
"dev.zio" %% "zio-test" % Version.zio % Test,
"dev.zio" %% "zio-test-sbt" % Version.zio % Test
)
)
.settings(buildInfoSettings("zio.zmx"))
.enablePlugins(BuildInfoPlugin)

lazy val coreJS = core.js
lazy val coreJVM = core.jvm

lazy val client =
crossProject(JSPlatform, JVMPlatform)
.in(file("client"))
.settings(
commonSettings,
crossScalaVersions := Seq(Version.Scala213),
stdSettings("zio.zmx.client"),
testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework"),
libraryDependencies ++= Seq(
"dev.zio" %%% "zio" % Version.zio,
"com.lihaoyi" %%% "upickle" % Version.upickle,
"dev.zio" %%% "zio-test" % Version.zio % Test,
"dev.zio" %%% "zio-test-sbt" % Version.zio % Test
)
)
.jvmSettings(
crossScalaVersions := Seq(Version.Scala213),
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % Version.zio,
"io.d11" %% "zhttp" % Version.zioHttp
),
run / fork := true,
run / javaOptions += "-Djava.net.preferIPv4Stack=true"
)
.jsSettings(
crossScalaVersions := Seq(Version.Scala213),
Test/run/javaOptions += "-Djava.net.preferIPv4Stack=true",
cancelable := true,
stdSettings("zio.metrics.connectors"),
libraryDependencies ++= Seq(
"com.raquo" %%% "airstream" % Version.airStream,
"com.raquo" %%% "laminar" % Version.laminar,
"io.laminext" %%% "websocket" % Version.laminext,
"io.github.cquiroz" %%% "scala-java-time" % "2.3.0",
"org.scala-js" %%% "scala-js-macrotask-executor" % "1.0.0",
("org.scala-js" %% "scalajs-test-interface" % scalaJSVersion % Test)
.cross(CrossVersion.for3Use2_13)
"dev.zio" %% "zio" % Version.zio,
"dev.zio" %% "zio-json" % Version.zioJson,
"dev.zio" %% "zio-streams" % Version.zio,
"io.d11" %% "zhttp" % Version.zioHttp,
"dev.zio" %% "zio-test" % Version.zio % Test,
"dev.zio" %% "zio-test-sbt" % Version.zio % Test,
),
scalaJSLinkerConfig ~= {
_.withModuleKind(ModuleKind.ESModule)
},
scalaJSLinkerConfig ~= {
_.withSourceMap(false)
},
scalaJSUseMainModuleInitializer := true
)
.settings(buildInfoSettings("zio.zmx.client"))
.settings(buildInfoSettings("zio.metrics.connectors"))
.enablePlugins(BuildInfoPlugin)
.dependsOn(core)

lazy val clientJS = client.js
lazy val clientJVM = client.jvm

lazy val examples =
(project in file("examples"))
.settings(
stdSettings("zio.zmx.examples")
)
.settings(
commonSettings,
publish / skip := true,
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % Version.zio,
"io.d11" %% "zhttp" % Version.zioHttp
)
)
.dependsOn(coreJVM)

lazy val benchmarks =
(project in file("benchmarks"))
.settings(
publish / skip := true
)
.enablePlugins(JmhPlugin)
.dependsOn(coreJVM)

lazy val docs = project
.in(file("zio-zmx-docs"))
.settings(
commonSettings,
publish / skip := true,
moduleName := "zio.zmx-docs",
publish / skip := true,
moduleName := "zio.zmx-docs",
scalacOptions -= "-Yno-imports",
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % Version.zio,
"io.d11" %% "zhttp" % Version.zioHttp
"io.d11" %% "zhttp" % Version.zioHttp,
),
ScalaUnidoc / unidoc / unidocProjectFilter := inProjects(root),
ScalaUnidoc / unidoc / target := (LocalRootProject / baseDirectory).value / "website" / "static" / "api",
ScalaUnidoc / unidoc / target := (LocalRootProject / baseDirectory).value / "website" / "static" / "api",
cleanFiles += (ScalaUnidoc / unidoc / target).value,
docusaurusCreateSite := docusaurusCreateSite.dependsOn(Compile / unidoc).value,
docusaurusPublishGhpages := docusaurusPublishGhpages.dependsOn(Compile / unidoc).value
docusaurusCreateSite := docusaurusCreateSite.dependsOn(Compile / unidoc).value,
docusaurusPublishGhpages := docusaurusPublishGhpages.dependsOn(Compile / unidoc).value,
)
.dependsOn(root, examples)
.dependsOn(root)
.enablePlugins(MdocPlugin, DocusaurusPlugin, ScalaUnidocPlugin)
Binary file removed client/js/ZIO.png
Binary file not shown.
28 changes: 0 additions & 28 deletions client/js/constants.ts

This file was deleted.

Binary file removed client/js/favicon.ico
Binary file not shown.
22 changes: 0 additions & 22 deletions client/js/index.html

This file was deleted.

Loading

0 comments on commit 23aa33a

Please sign in to comment.