Changes

Summary

1. wait for kafka client consumer and producer actors initialization to be sure any authentication errors are handled with backoff (details)
2. wait for success messages in tests (details)
3. fix mock mapper (details)
4. make init timeout configurable for kafka producer and consumer (details)
5. implement support for wildcard (*) in selected fields for RetrieveThing, SudoRetrieveThing and RetrieveFeatures (details)
6. fix unit test (details)
7. review: (details)
8. document not to use '*' as feature ID in basic-feature.md; (details)
9. Avoid passing the context in AcknowledgementForwarderActorStarter (details)
10. fixed/simplified using multiple MappingProcessors via construct "concatLazy(Source.lazySource())" with a simpler and more efficient operation (details)
11. Replace client actor refs refreshing mechanism (details)
12. document field selector with wildcard and extraFields with wildcard; (details)
13. Add counter for connectivity acks to measure messages waiting for acks in a given moment (details)
14. configured the default "processorPoolSize" of new connections to 1 (details)
15. fixed ImmutableConnectionTest (details)
16. introduce caffeine cache to cache oauth tokens for http push connections (details)
17. rephrase docu; (details)
18. only sync actors with aggregation actor for more than one client (details)
19. stabilize tests for expiring JWTs (details)
20. use http-push dispatcher for requests to token endpoint (details)
21. move common functionality of expanding feature id wildcard to ThingFieldSelector class and adapt code where the functionality is used (details)
22. Allow to disable automatic passivation for specific shard regions (details)
23. extend selectable fields with "__lifecycle"; (details)
24. Issue #1273: Deal with unavailable Kafka broker in KafkaPublisherActor. (details)
25. moved feature id wildcard expansion logic into ThingsModelFactory; (details)
26. Issue #1273: Moved logic for escalating connection failure to avoid unwanted side effects. (details)
27. fix that very actively updates things indexed in the search could have a wrong search index (details)
28. improve documentation about OpenId connect issuer configuration (details)
29. Add sync between pubsub ddata and cluster state. (details)
30. Add unit test for cluster state sync. (details)
31. Fix license header year. (details)
32. make HealthSignal serializable by implementing marker interface AkkaJacksonCborSerializable (details)
33. Make ClientActorRefs serializable so we can send this directly instead of (details)
34. Add serialization test for ClientActorRefs (details)
35. Simplified serialization test a bit. (details)
36. Add a random delay to ThingUpdater force-update-after-start-timeout to avoid load spikes. (details)
37. * update source and target level to 17, update maven-plugins and dependencies. (details)
38. review: just minor formating / javadoc fixes (details)
39. Issue #1283: Use JDK 17 for github actions build (details)
40. #1283: use jdk17 docker base images (details)
41. fix infinite loop (details)
42. use ThreadSafeDittoLogger for KafkaMessageTransformer; (details)
43. add AcknowledgementAggregatorActor to EXCLUDED_ADDRESS_REPORTING_CHILD_NAME_PATTERN to avoid warning logs; (details)
44. enabled ditto-placeholders japicmp comparison (details)
45. prepare Ditto 2.3.0 release and release notes, blogpost, etc. (details)
46. set next "binary-compatibility-check.version" to released 2.3.0 (details)
47. add content-type to very beginning of protocol-specification.md (details)
48. Test and fix false positives of ClusterStateSyncBehavior due to unserialized local actor refs. (details)
49. fix that placeholder `time:now` did not resolve to the current time when used in a connection's headerMapping (details)
50. ThingUpdater: execute force-update-after-start without waiting for the next event. (details)
51. Add pipeline execution for array values and introduce fn:split() function (details)
52. Allow to use fn:filter() on previous pipeline elements (details)
53. use all available placeholders for HttpPushValidator and AbstractMqttValidator like in the other validators; (details)
54. replace deprecated Akka config "passivate-idle-entity-after" with "passivation.strategy" (details)
55. remove requesting an acknowledgement for UpdateThing command and do not report inconsistent things in search health status (details)
56. add test cases for entity and connection id placeholder in Mqtt3ValidatorTest and HttpPushValidatorTest; (details)
57. remove obsolete unit test, functionality was removed (details)
58. Add filter for search index diff updates on revision. Trigger force update on mismatch. (details)
59. Search update: Remove unused imports; remove errors from missed update detection. (details)
60. fixed that JSON `null` in "correlation-id" of Ditto Protocol headers were parsed as JSON String `"null"` (details)
61. Add counters to track various kinds of search updates. (details)
62. fixed that CommandAndCommandResponseMatchingValidator.validateCorrelationIdsMatch also match when the correlation-id of a commandResponse starts with the correlation-id of the command (details)
63. optimized signal header setting logic (details)
64. Add feature toggle for force-update-after-start. (details)
65. Add OR operator to fn:filter(like) operator (details)
66. review: minor style fixes; removed non-needed imports (details)
67. Refactory away an AtomicBoolean in ResponseReceiverCache. (details)
68. fix that creating policy was wrapped in "ask with retry" setup - this is wrong as create is idempotent (details)
69. added Ditto release notes for version 2.3.1 (details)
70. Remove left-over tagging for waiting ack counter (details)
71. Ignore DittoMessageMapper for hono delivery failed notifications (details)
72. isPatchUpdate would always be false (details)
73. ThingDeleted event did not result in ThingDeletedModel (details)
74. Don't expect a writeModelCount for ThingDeleteModel (details)
75. Add test validating KafkaProducerStream restart on error. (details)
76. bugfix: fixed that LiveSignalEnforcement.handleAskTimeoutForCommand() threw a 503 (ThingNotAvailable) exception and caused a race condition against a "graceful timeout" (details)
77. Add tests for filtering of expired inbound Kafka messages (details)
78. Add HttpPushSpecificConfig to enable custom idle timeout configuration per connection (details)
79. Add Switch to Java 17 to ditto release notes (details)
80. fn:split() review: (details)
81. document 'is-group-topic' and 'aggregate' header for piggyback commands; (details)
82. set "is-group-topic" to true for some piggyback command examples; (details)
83. fn:split() review: (details)
84. fn:filter() review: adjusted signature text to state that first parameter is optional (details)
85. review: minor code formatting changes; (details)
86. first check for incorrect patch update in the if branch; (details)
87. added documentation for OpenId Connect "jwt:" placeholder and new "fn:split()" function applicable for that placeholder (details)
88. Add existing HTTP Push specific config items to HttpPushSpecificConfig (details)
89. made defaults for "max-source-number" and "max-target-number" for connections explicit in config (details)
90. review: adjusted/fixed log templates (details)
91. Add comma (details)
92. added release notes for Ditto 2.3.2 (details)
93. added release notes for Ditto 2.3.2 to sidebar (details)
94. update mutability detector and re-enable all mutability tests (details)
95. fixed that search updater did too much patch failures due to recovery bug (details)
96. reduce receiveTimeout to 5 seconds after a ThingDeleted event was consumed in ThingUpdater to quickly shut it down (details)
97. enhanced test by asserting for the recovered write model (details)
98. review: (details)
99. adapt deployments to java 17 hotspot (details)
100. fix dockerfile-release (details)
101. Add some description to more complex DittoJwtAuthorizationSubjectsProviderTest cases (details)
102. Also update thingRevision and policyRevision on recovery of ThingUpdater (details)
103. Add documentation on fn:filter RQL functions (details)
104. fix compilation error (details)
105. Remove custom JAVA_OPTIONS in favor of standardized JAVA_TOOL_OPTIONS (details)
106. Consider devops request as authorized if no subjects are expected (details)
107. added more Grafana dashboards as templates (details)
108. delete replaced Hot_Entities.json dashboard (details)
109. install wget during setup (details)
110. remove new line at last command (details)
111. Enhance logging for mapper instantiation (details)
112. Enhance logs for mapper registration (details)
113. Fix some tests for build on windows and raise version for maven deploy plugin (details)
114. remove test jar from connectivity service module (details)
115. configure classesDirectory for failsafe plugin explicitly for connectivity service (target/classes directory is missing in test classpath otherwise) (details)
116. use same version as defined in the parent pom (details)
117. fixed that ThingUpdater was already stopped when processing "ThingDeleted" events with a consistency-lag > 5 seconds (details)
118. update prerequisites in order to build Ditto (details)
119. added license headers to run config files (details)
120. improved logging for search "bulkWrite": show included correlation-ids in a bulkWrite DEBUG statement when debug logging is activated (details)
121. improved logging statement to include thingIds + BSON filter as well (details)
122. fixed potential NPE (details)
123. fixed some method names and log statements (details)
124. Remove unnecessary option (details)
125. Start policies service at first to make sure cluster can be built as soon as possible (details)
126. Adapt documentation for building ditto (details)
127. added possibility to add JVM options for e.g. specifying "--add-opens" for ditto-connectivity required with Java 17 (details)
128. simplify maven.yml GitHub action: (details)
129. updated legal information after updating lots of dependencies for Java 17 (details)
130. fix using environment vars in docker-nightly.yml (details)
131. fixed JVM_CMD_ARGS invalid quotation (details)
132. added "restart" policy "always" to docker-compose (details)
133. added platform linux/arm/v7 in nightly image build (details)
134. fixed/improved JVM settings of sandbox + example k8s descriptor files (details)
135. fixup some READMEs in deployment section, adding system requirements (details)
136. remove "linux/arm/v7" from supported platforms in nightly docker build again (details)
137. update jruby used in maven-gem-plugin to latest 9.3.3.0 in order to get it running with Java 17 JDK (details)

wait for kafka client consumer and producer actors initialization to be sure any authentication errors are handled with backoff

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>

wait for success messages in tests

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>

fix mock mapper

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>

make init timeout configurable for kafka producer and consumer

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>

implement support for wildcard (*) in selected fields for RetrieveThing, SudoRetrieveThing and RetrieveFeatures

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

fix unit test

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

review:
minor code format changes and readability improvements;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

document not to use '*' as feature ID in basic-feature.md;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

Avoid passing the context in AcknowledgementForwarderActorStarter

* The sender could be some different when actually starting the forwarder
  actor. Therefore directly pass the sender at the time we know it

Signed-off-by: Yannic Klem <Yannic.Klem@bosch.io>

fixed/simplified using multiple MappingProcessors via construct "concatLazy(Source.lazySource())" with a simpler and more efficient operation

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Replace client actor refs refreshing mechanism

* Former the refreshing was started by all clients. The way it happened
  resulted in the problem that dead client actors are never removed from
  the client actor refs
* Now the refreshing is triggered by the ConnectionPersistenceActor which
  provides a SourceRef of all client actor refs (including themselfs) to
  every client actor

Signed-off-by: Yannic Klem <Yannic.Klem@bosch.io>

document field selector with wildcard and extraFields with wildcard;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

Add counter for connectivity acks to measure messages waiting for acks in a given moment

Signed-off-by: David Schwilk <david.schwilk@bosch.io>

configured the default "processorPoolSize" of new connections to 1
* fixed some openapi errors

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

fixed ImmutableConnectionTest

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

introduce caffeine cache to cache oauth tokens for http push connections

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

rephrase docu;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

only sync actors with aggregation actor for more than one client

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>

stabilize tests for expiring JWTs

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

use http-push dispatcher for requests to token endpoint

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

move common functionality of expanding feature id wildcard to ThingFieldSelector class and adapt code where the functionality is used

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

Allow to disable automatic passivation for specific shard regions

Signed-off-by: Yannic Klem <Yannic.Klem@bosch.io>

extend selectable fields with "__lifecycle";
fixed test cases in ThingPersistenceActorTest;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

Issue #1273: Deal with unavailable Kafka broker in KafkaPublisherActor.
Until now a Ditto Kafka connection with targets only did not cope properly with the Kafka broker becoming unavailable.
In such cases, publishing timed out and produced appropriate entries in connection log.
Connection live status however, remained always 'open'.
With this commit, a TimeoutException while publishing is handled in a way that a ConnectionFailure is propagated to parent BaseClientActor.
This sets connection live status to 'misconfigured' and triggers reconnect with back-off semantic.
Thus, a potentially unavailable Kafka broker gets appropriately reflected by connection status.

Signed-off-by: Juergen Fickel <juergen.fickel@bosch.io>

moved feature id wildcard expansion logic into ThingsModelFactory;
removed "__lifecycle" field from SELECTABLE_FIELDS in ThingFieldSelector;

Signed-off-by: Stefan Maute <stefan.maute@bosch.io>

Issue #1273: Moved logic for escalating connection failure to avoid unwanted side effects.

Signed-off-by: Juergen Fickel <juergen.fickel@bosch.io>

fix that very actively updates things indexed in the search could have a wrong search index
* this can e.g. occur during rolling update for very often updated things due to shard reallocation to another cluster member
* simply force an update a configurable amount of time after a ThingUpdater was started
* changed the default for "automaticPassivationEnabled" in ShardRegionFactory.createShardRegion to "false" as this also conflicts with the manual passivation strategy of search ThingUpdater shard region

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

improve documentation about OpenId connect issuer configuration

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Add sync between pubsub ddata and cluster state.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Add unit test for cluster state sync.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Fix license header year.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

make HealthSignal serializable by implementing marker interface AkkaJacksonCborSerializable

Signed-off-by: Dominik Guggemos <dominik.guggemos@bosch.io>

Make ClientActorRefs serializable so we can send this directly instead of
sending a SourceRef

Signed-off-by: Yannic Klem <Yannic.Klem@bosch.io>

Add serialization test for ClientActorRefs

Signed-off-by: Yannic Klem <Yannic.Klem@bosch.io>

Simplified serialization test a bit.

Signed-off-by: Juergen Fickel <juergen.fickel@bosch.io>

Add a random delay to ThingUpdater force-update-after-start-timeout to avoid load spikes.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

* update source and target level to 17, update maven-plugins and dependencies.
* ignore all tests using MutabilityDetector for now, because it is not yet compatible with java 17.
* fix javadoc headings.

Signed-off-by: Johannes Schneider <johannes.schneider@bosch.io>