Data Stream Connector (Kafka, RabbitMQ, Azure Service Bus, etc.)

by
Odoo

862.08

v 13.0 Third Party
Availability
Odoo Online
Odoo.sh
On Premise
Odoo Apps Dependencies Discuss (mail)
Community Apps Dependencies
Lines of code 4021
Technical Name data_stream_connector_module
LicenseOPL-1
Websitehttps://www.solvti.pl
You bought this module and need support? Click here!
Availability
Odoo Online
Odoo.sh
On Premise
Odoo Apps Dependencies Discuss (mail)
Community Apps Dependencies
Lines of code 4021
Technical Name data_stream_connector_module
LicenseOPL-1
Websitehttps://www.solvti.pl

WE SOLVE
REAL
BUSINESS PROBLEMS

Odoo Data Stream Connector

Seamless data exchange between different Odoo instances or between Odoo and external systems using data streams.

This module allows you to exchange data with any system that can read or send data to a data stream. This module is designed to be flexible, and it can be used to suit a variety of business use-cases. For instance, you can use it to publish events on a data stream when a record is added or modified in your Odoo instance, and to send changes to a data stream for any chosen data model.

The module includes base logic for publishing and reading changes/events, and we've included providers for well-known data streams such as RabbitMQ and Confluent Kafka. If you use a different data stream technology, you can write your own provider or reach out to us to implement one for you.

While the setup may require technical expertise, our module includes detailed technical documentation, which can guide you through the installation and setup process. However, if you require assistance, our team of experts is available to help you get started and answer any questions you may have.

For high-volume data publishing, we recommend using our free module that integrates Data Stream Connector with OCA/queue. This module prevents your Odoo instance from slowing down, making data publishing faster and more efficient

  Requirements and limitations

  • Works with Odoo Community and Odoo Enterprise.
  • You need to have your a data stream to use this module.
  • Currently supported data streams: RabbitMQ and Confluent Kafka. To use other data streams, a provider must be developed.
  • Currently it is needed to make rules configuration in code for your model. Not everything is possible from the user interface yet. See: documentation bellow.

  Resources

What is Data Streaming:
https://www.confluent.io/learn/data-streaming/

Apache Kafka:
https://kafka.apache.org/

Confluent Kafka - Enterprise-grade distribution of Apache Kafka:
https://www.confluent.io/

RabbitMQ:
https://www.rabbitmq.com/

Integrate multiple Odoo instances

With Odoo Data Stream Connector, you are able to exchange data with other Odoo instances.

Example image

You will set-up which data models you want to send to the data stream and on the other Odoo instance, you will set-up which data you will download from your data stream. It's also possible to synchronize all data entries or only some.

  Example business use cases

  • One Odoo instance is source of truth for Employee Data, and you need to send it to other Odoo instances.
  • Two companies that both use Odoo want to mirror some Projects between Odoo instances.
  • You have a small number of employees who need Odoo Enterprise, and the majority are fine with Odoo Community. So you have two instances, but want to share some data between them.
  • You're migrating from an old Odoo version to a new one, and will run your business on two Odoo instances in one year, but want to keep some data in synch.

Integrate Odoo with 3rd party applications

If you use Odoo in a larger digital ecosystem, chances are quite high, that you would like to integrate data of Odoo with either other custom made applications or off the shelf programs.

Odoo connection to 3rd party apps

Especially in ecosystems, where many systems must exchange data, it makes sense to publish changes to data streams, so that you don't have to develop new integrations each time you need some data. You can simply connect to a data stream.

  Example business use cases

  • You want to exchange data between Odoo and your dedicated application written in any technology.
  • You have a large fleet and you want to use a dedicated fleet software that has many more features that fleet module in Odoo. You want to download fleet data into Odoo. It's important to check with the software vendor if they support publishing changes to data streams.

Migrations

You can use Data Stream Connector also for migrations.

Odoo connection to 3rd party apps

With one button, you can send all selected data from Odoo to a data stream and then download/consume the data in any point in time.

  Example business use cases

  • You want to merge two Odoo instances. You can not simply add all the data from one database to another. But you can select what data you want to export from one Odoo instance to the data stream and later download only this data.
  • You want to update your Odoo to a newer version. But as you made a lot of customizations, it would be very costly to do a database upgrade. Instead, you can migrate the code, run the old Odoo and the new one simultaneously and export only the data you need to the new Odoo.

Connect to multiple data streams

Data Stream Connector is built to accommodate various data stream technologies such as Kafka, RabbitMQ, Azure Bus, and Google Cloud Pub/Sub. However, a specific provider needs to be developed to use the module with a particular data stream

Publish changes to a stream

This module publishes data stream events when a record is created or modified in your Odoo instance. You can select the data model for which changes are sent to the data stream

Delay publication of changes

We've integrated our open-source module with OCA/Queue to publish data stream changes without affecting your Odoo's performance

Data parsing

You can choose to publish raw field values or apply different parsers to each field before sending data to the data stream

Field mapping

The Data Stream Connector module offers a user-friendly interface for selecting which fields to exchange and mapping fields between data streams and Odoo

Receive data in batches

Our Data Stream Connector module enhances performance by enabling you to define batch sizes and process data in batches per data stream

Works even with relational fields

Data Stream Connector allows you to exchange not only simple fields such as char, text, and int but also relational fields. You can choose a mapping option for each relation

Send data manually

You can manually send data to a data stream, which is particularly useful when initiating the integration. With this feature, you can use Odoo's domain feature to select which data to send

Support

We offer 90 day support from the date of the purchase to every app.

Submit a ticket

WE SOLVE REAL BUSINESS PROBLEMS

We also offer

Odoo Implementation

Custom Development

Support Services

Who we are


Solvti is a software company based in Wroclaw, Poland, with customers across the whole Europe. We're a certified Odoo Partner, with projects in both Odoo Enterprise and Odoo Community.

Solvti was founded in 2021 with the mission not only to deliver software, but to solve the real business problems.

There's so much more we can do

Go on, try us

Contact us

DATA STREAM CONNECTOR

Description

Base module for cross-system data exchange by using streaming. Implementation of Instances, Rules and Field Mappers.

Table of contents

Installation

  • You can install all features by installing data_stream_connector_module.
  • If you wish to use delay feature, make sure you have OCA Job Queue, as well as our module data_stream_queue_job installed and configured.

Usage

To use this module, you need to:

  1. Activate Developer Mode.
  2. Go to Settings and click Odoo Data Stream Connector menu.
  3. Create Instance with relevant provider settings.
  4. Add Credentials for your provider.
  5. Create Data Stream Rule for your model.
  6. Activate Scheduled Action for receiving the data or use "Manually Send Data" button for initial data sending.

Configuration

Developers

Basic Configuration

  1. More configuration may be required for specific provider. See more in READMEs.
  2. Configure all required Credentials for your provider using System Parameters or environment variables.
  3. Add data stream instance and select your provider.
  4. Add data stream rules for your instance.

Developer

  • Add Data Stream Mixin ("data.stream.mixin") to a data model, for which you want to activate the exchange functionality.
class ResPartner(models.Model):
  _name = "res.partner"
  _inherit = ["res.partner", "data.stream.mixin"]
  • You can use our solution for auto-generating rules using XML function and attribute in class with fields you wish to use in data streaming.
  • To do so, add class attribute as list of tuples: [("field_name","fieldNameInPayload", None)]. There you can define Data Stream Rules with Mappers fields. Example use case:
_data_stream_rule_fields_send = [
  ("name", "name", None),
  ("category_id", "category_id", None),
  ("company_id", "company_id", None),
  ("bank_ids", "bank_ids", None),
]

_data_stream_rule_fields_receive = [
  ("name", "name", None),
  ("category_id", "category_id", None),
  ("company_id", "company_id", None),
  ("bank_ids", "bank_ids", None),
]
  • Use method auto_create_rules() in order to create rules automatically. You can also write your own method for that.
  • Add this method to XML function to generate rules during module installation.
<function model="res.partner" name="auto_create_rules">
     <value eval="[ref('base.partner_root')]" />
     <value eval="True" />
     <value>send</value>
     <value>your_module.your_data_stream_instance</value>
 </function>
 <function model="res.partner" name="auto_create_rules">
     <value eval="[ref('base.partner_root')]" />
     <value eval="False" />
     <value>receive</value>
     <value>your_module.your_data_stream_instance</value>
 </function>
  • If you do not define process for unlink, mixin will use standard Data Stream unlink behavior, which is:
_unlink_option = "restrict"  # Default
  • If you wish to allow users to delete records, we recommend to use soft-delete.
_unlink_option = "soft-delete"
  • See more unlink option in /data_stream_base/models/data_stream_mixin.py UNLINK_OPTION_TYPE.

Send Option

You can use exchange_uid field in two ways:

  • Use Odoo standard mechanism: xml_id (Default)
  • Use Hashlib to create a unique Exchange UID hash.

To be able to use second option you will need to add this into your model where you added mixin:

def _sanitize_exchange_uid(self, use_hash=False):
  return super()._sanitize_exchange_uid(use_hash=True)
  • More configuration options can be found in a particular module's README file.

Known issues / Roadmap

  • two-way sync is not supported. To achieve such synchronization, you will need to create 2 rules separately. One for sending and one for receiving.

Changelog

13.0.2.0.0

  • added: Implementation of Data Stream version 2.
  • added: New 4 options for deleting records which were already send by data stream connector.
  • added: Global auto_create_rules method to create rules through xml files.
  • changed: Improved sending mechanism.
  • changed: Improved receiving mechanism.
  • changed: active field is not longer required in data stream connector.
  • changed: False value in send mechanism will be present only for Boolean fields.
  • fixed: relations search mechanism.
  • removed: Additional Rules.

13.0.1.0.2

  • changed: Updated README, Description
  • changed: Refactors

13.0.1.0.1

  • added: Possibility to use delay feature.

13.0.1.0.0

  • Initial implementation of Data Stream Connector modules.
Odoo Proprietary License v1.0

This software and associated files (the "Software") may only be used (executed,
modified, executed after modifications) if you have purchased a valid license
from the authors, typically via Odoo Apps, or if you have received a written
agreement from the authors of the Software (see the COPYRIGHT file).

You may develop Odoo modules that use the Software as a library (typically
by depending on it, importing it and using its resources), but without copying
any source code or material from the Software. You may distribute those
modules under the license of your choice, provided that this license is
compatible with the terms of the Odoo Proprietary License (For example:
LGPL, MIT, or proprietary licenses similar to this one).

It is forbidden to publish, distribute, sublicense, or sell copies of the Software
or modified copies of the Software.

The above copyright notice and this permission notice must be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.

Please log in to comment on this module

  • The author can leave a single reply to each comment.
  • This section is meant to ask simple questions or leave a rating. Every report of a problem experienced while using the module should be addressed to the author directly (refer to the following point).
  • If you want to start a discussion with the author or have a question related to your purchase, please use the support page.