spotify / heroic Archived

* heroic-730 Implement new Bigtable timeout & retry settings

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* heroic-733 lowered timeouts, added more block comments.

- passing all tests
- next is to refactor the 5 parameters into a class

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* heroic-733 updated failing unit test

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* heroic-733 attempting refactor connection settings into one class

so that they're not repeated several times over

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* updated timeout settings as agreed with @SergeyR on Slack

* heroic-733 small settings & comment corrections

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* heroic-733 small settings & comment corrections

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixed runtime IT error in console & syntax warnings

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* added tagValuesTruncatedSuggestMany to exercise difference between CI

and localhost IT test runs where it works locally but not in CI.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixed checkstyle error.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixed checkstyle error.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixes logging in unit & integration tests.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixes inconsistent tagValuesTruncatedSuggest behaviour

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* made the tests in *SuggestBackend*IT.java deterministic

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* increased time available for deleteSeries() to complete

down the road all tests of this kind (timer-based) will
need redoing as they're lame and brittle.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* refactored 6 retry & timeout settings into new POD class

which is then added as a field to BigtableBackend, Module etc.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* reduced cohesion between modules by switching implementations

from MetricsConnectionSettingsModule to MetricsConnectionSettings

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* moved MetricsConnectionSettingsModule to Bigtable module since...

that's where it belongs, basically.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* *actually* moved MetricsConnectionSettingsModule to Bigtable module

since...that's where it belongs, basically.

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* added {@link ...} javadoc

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* fixed comment

Signed-off-by: Peter Kingswell <peterk@spotify.com>

* implemented review feedback

- removed useless/confusing @JSON annotations
- changed fields from Integer to int
- also fixed code analysis warnings

* Rename .java to .kt

* PR feedback - refactored POD BT Java class to Kotlin

* implemented a workaround for @Inject

not working as documented (seemingly)

* improved comments, removed unnecessary subclass

* io.grpc -> 1.35.0 & bigtable-client-core -> 1.19.0

* increasing heroic system test startup time by 30s

to prevent the intermittent timeouts that are being observed.

* fixed Could not find policy 'pick_first' exception.

- occurred when a timeout happened
- full message: java.lang.IllegalStateException: Could not find policy
'pick_first'. Make sure its implementation is either registered to
LoadBalancerRegistry or included in
META-INF/services/io.grpc.LoadBalancerProvider from your jar files.

* adds integration test for Bigtable timeouts

* report the failing test's actual content

* fix testBackendTimesOutCorrectly assertion

* unit test fix attempt #2

* set Google-default timeout and retry

settings.
tidied up docs too

… x-client-id (#763)

* WIP BT timeout logging and metrics

* commented new methods

* increase timeout to get CI passing

* retired reportClientIdSuccessCount + refactorings

* refactoring + improved comment

…ut (#749)

* Remove static import statement that keep failling sporadiclly

* Remove redundant class Data.java

this is the implementation of x-client-id header enforcement in Heroic (as opposed to the just-reverted Envoy implementation)

* dynamic read index count if config flag true

* relocate config flag, prevent bad case of single mapping && dynamic

* create helper function and add test

* make maxReadIndices configurable for proper delete support. further bifurcation of index mapping classes

* add docs

* Add distribution query integration test. (Part 2 of 2)

* Addresses PR comments

* Addresses PR comments

* Add a new group combiner that handles tdigest query without aggregation

* Addresses pr comments and add boilerplate methods to metricollection to handle tdigestPoint creation

* include mapping for metadata index interval

* formatting

This adds error flags and exception messages to the spans for
metadata, suggest, and bigtable writes when the chain of futures
fails. The same exceptions are also logged. Previously some
exceptions, such as grpc errors, could be masked by sl4j settings.

The trace for a metric write was cleaned up to remove several
intermediary spans. These spans did not have useful information and
had no branching paths, so they were just clutter in the overall
trace.

Closes #724.

* Add distribution query integration test. (Part 2 of 2)

* Addresses PR comments

* Addresses PR comments

… of 2) (#734)

* Moves tdigest stat computation to sharedGroups aggregation phase.
This PR adds TdigestPoint, a datastructure  used during the last phase of aggregation.
Unlike distributionPoint, tdigestPoint value are un-serialized tdigest.

* Rename TdigestStatInstanceUtils and add copyright statement

…e test (#731)

* add Series getter for hashCode, modify write cache conditions, update tests

* eliminate dup code

* Ensure metrics that are explicitly set to zero are not dropped

* Fixes a typo in distribution column family name

* Add distributionPoint aggreagation components

* Add tdigest percentile computation and thread safety

* Remove failed test because it depends on module registration which is not included in the PR

* Add javadoc to new classes

* Remove unused import statement

* add config to optionally index RIs as tags, modify metadata request object to include RIs as tags

* add provides method, fix codestyle

* trace value as double

* adjust implementation to use less mem

* properly add RI keys

* modify format for writing resource identifiers to elasticsearch, revise tests

* duplicate tests, new metadata backend and config for index RI tests

* codestyle

* do not write resource key to elasticsearch

* use appendData method for repeated code

* Ensure metrics that are explicitly set to zero are not dropped

* Fixes a typo in distribution column family name

Co-authored-by: Lauren Muhlhauser <lmuhlha@gmail.com>

Closes #716.

HQL has been stable for quite some time. Depending on the result of
481, we may decide to replace HQL. But for now it is disingenuous to
list it as experimental.

* Remove cleaning of QuotaWatchers map; wrap main call to exception

… distribution from bigTable. (#700)

* Add distribution support to spotify100_proto and BigtableBackend writer

* Add distribution support to spotify100_proto and BigtableBackend writer

* Remove message type from spotify_100.proto

* Add distribution support for spotity100 and BigTbableBackend Read/Write

* Add distribution support for spotity100 and BigTbableBackend Read/Write

* Add Javadoc comments to JsonMetric and code reformatting

* Remove empty line from PubSubconsumerIT

… writer (#699)

* Add distribution support to spotify100_proto and BigtableBackend writer

* Add distribution support to spotify100_proto and BigtableBackend writer

* Remove message type from spotify_100.proto

* Add global data points stats

…tify:dns (#697)

Co-authored-by: Samantha Fadrigalan <samanthanoellef@spotify.com>

* Fix type String? to CharSequence type mismatch

* Remove use of depracated Metric functions and change param names to follow overridden AbstractBucket method param

* Remove use of deprecated bt ColumnFamilies getter

* Suppress depracation warning: function is already using recommended ResolvableFuture#fail(Throwable)

* Use recommended way of creating NamedWatch

Co-authored-by: Samantha Fadrigalan <samanthanoellef@spotify.com>

* Add total stats across all queries in local manager
* Temporarily changed test emulator docker image