Changes

Issue #878: [WIP] add local and remote data structures for declared ack labels with group.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: [WIP] add AckSupervisor; test AckUpdater (failing)

- akka.japi.Pair is not serializable. Use a different data structure.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

don't use DittoLoggerFactory to get logger in AbstractStreamingActor in order to prevent scala/java/javadoc compile errors due to compile circular dependencies within this module

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: switch AckUpdater to LiteralDData for ease of serialization; test remote conflict.

- Delete AckDData, which uses a non-serializable type akka.japi.Pair
  as value of ORMultiMap.

- Opted to use String as value type of ORMultiMap and deserialize
  on each ddata change notification. This way, sweeping changes to
  serialization configuration in all dependent services are avoided.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: test & fix a local conflict case for AckUpdater; add javadoc.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: moved inner classes of AbstractUpdater into own "API" package
* made abstract classes package private where now feasible
* small javadoc adjustments

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: Replace AcksUpdater by AckUpdater.

- Add serialization and deserialization of grouped acknowledgement
  label declarations.

- Add typed change notifications from AckUpdater.

- Handle remote racing.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: move AckUpdater messages to package /api.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: replace GroupedAckLabels by Grouped for use in topics ddata.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: hardcode hash family size to 2; switch compressed topic type from ByteString to Long to prepare for JSON serialization.

Issue #878: add group to topics ddata; remove incrememental update.

Incremental update removed due to losing granuality of topics
in the distributed data.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: add command to publish signals with groups.

Command deserialization is extended to give recursive access
to the registry in order to deserialize inner signals.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Restrict published messages to signals.

Reason: Serialization of PublishSignal command.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Fix serialization of PublishCommand.

Inner signal may not be a command. Should recurse onto mapping
strategies level instead of the same (command) registry.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Fix parsing of non-recursive jsonifiable..

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: distribute signals according to groups on the publisher side.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: distribute published messages according to groups at the subscriber.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: add methods to declare ack labels and subscribe in a group.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: restrict weak acks to chosen groups; fix ConnectionPersistenceActorTest; fix that grouped ack label declarations conflict with themselves.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: make client actors aware of each other; fix StreamingSessionActorHeaderInteractionTest.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: move auth and acknowledgement forwarding from ConnectionPersistenceActor to OutboundDispatchingActor; add InboundSignal; fix deserialization of null event metadata.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: turn DittoProtocolSub into an actor system extension.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: fix dropping of search events due to same origin.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: fixed just some minor cosmetic final/import stuff

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: add random gossip between client actors to recover from message loss; make client actors watch each other to remove zombies from cache.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: renamed methid PublisherIndex.allot... to assign...

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: move event subscription from connection persistence actor to client actors.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: add javadoc for the group-size relation of PublishSignal.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Remove the detour of ThingSearchCommand via connection persistence actor where possible.

Client actors now dispatch search commands to each other whenever
they contain a subscription ID to identify the intended recipient.
Only CreateSubscription is routed through the connection persistence
for round-robin subscription ID assignment.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Make client actors talk to the local proxy actor only.

This removes an unnecessary remoting from client actor node to
the connection persistence actor node.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: removed no longer used method DistributedSub#subscribeWithAck

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: add and test pubsub recovery behavior.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: fix javadoc errors.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: migrate connection persistence actor tests & fix them.

- Tests related to event publishing moved to OutboundDispatchingActorTest.

- Test about search command dispatching moved to BaseClientActorTest.
  Fixed an issue where a portion of the client actors are not getting
  any search command forwarded due to hashing.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: minor naming stuff; Sonar fix

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: review: not issue-related cleanup stuff

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: add config option to start all client actors on the same node.

For single-connectivity instance deployments, starting multiple client
actors on the same node increases concurrency and reduces the impact
of latency on connection throughput.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Fix various TODOs; add test for invalid subscription ID.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Use connection ID as client ID if the connection has 1 client.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: fix "refreshAfterWrite" exception with loader.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Replace WRITE_ALL consistency of topics ddata by optimistic waiting.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: update ddata metrics; remove useless params and mocks.

DData topic count metrics replaced by size estimation due to
introduction of literal group names into the ddata of topics
and of declared acknowledgement labels.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: remove unused class LiteralSubscriptions.

Class was used only by an unnecessary interface method.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: re-introduce diff-based ddata updates.

Akka does not attempt to compute the diff of successive ddata updates,
which is reasonable because Akka does not know that all updates of an
ORSet comes from the same node. Therefore Ditto should compute the diff
so that no unnecessary replication occurs.

Consequently, replicators no longer emits Replicator.Changed events
every update cycle because unchanged subscriptions cause no update.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Make replication delay estimation less pessimistic.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

 Issue #878: Make replication delay estimation more pessimistic; fix a double assignment in AckSupervisor.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: [wip] try reverting to write-consistency-based SubAck for topic subscriptions

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Move all pubsub-related classes to the same module.

The following modules no longer depends on ditto-services-models-concierge:

- ditto-services-gateway-streaming
- ditto-services-connectivity-messaging

Gateway and connectivity starter modules still depend on concierge service
models because they need ConciergeForwarder.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Revert 20c64713; make ddata replication delay more pessimistic.

This reverts commit 20c64713f0fbda1fee6ad93facdbeba9c8ca6ff3.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: Fix handling of Terminated messages by Pub- and AckSupervisor.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: adjusted javadoc in Subscriptions

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: review: renamed LiteralUpdate.replaceAll method to withInserts; fixed toString()

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: review: fixed calculation of "change" in AbstractSubscriptions
* adjusted Subscribe, AbstractRequest Signatures to accept a Collection instead of a Set in order to make the "topics" guaranteed immutable w/o creating too much Set copies

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

fixed logging of exception in AmqpPublisherActor

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: replace instance ID by default client ID in BasePublisherActor.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: stabilize MQTT client actor tests; remove unused metric.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: configure pubsub with a small ddata notify-subscribers-interval for responsiveness.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>

Issue #878: review: removed "final" from interface method

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: review: replaced "-" separator of connectionId and random UUID for "clientId" with "_" in order to be able to easily split 2 UUIDs

Signed-off-by: Thomas Jaeckle <thomas.jaeckle@bosch.io>

Issue #878: Reorder CreateConnection stages to stabilize ConnectionPersistenceActorTest.enabledConnectionLogsAreEnabledAgainAfterModify.

Signed-off-by: Yufei Cai <yufei.cai@bosch.io>