Data Stream Connector (Kafka, RabbitMQ, Azure Service Bus, etc.)
by Solvti Sp. z o.o. https://www.solvti.pl$ 862.08
Availability |
Odoo Online
Odoo.sh
On Premise
|
Odoo Apps Dependencies |
Discuss (mail)
|
Community Apps Dependencies | Show |
Lines of code | 4021 |
Technical Name |
data_stream_connector_module |
License | OPL-1 |
Website | https://www.solvti.pl |
Availability |
Odoo Online
Odoo.sh
On Premise
|
Odoo Apps Dependencies |
Discuss (mail)
|
Community Apps Dependencies | Show |
Lines of code | 4021 |
Technical Name |
data_stream_connector_module |
License | OPL-1 |
Website | https://www.solvti.pl |
WE SOLVE
REAL
BUSINESS PROBLEMS
Odoo Data Stream Connector
Seamless data exchange between different Odoo instances or between Odoo and external systems using data streams.
This module allows you to exchange data with any system that can read or send data to a data stream. This module is designed to be flexible, and it can be used to suit a variety of business use-cases. For instance, you can use it to publish events on a data stream when a record is added or modified in your Odoo instance, and to send changes to a data stream for any chosen data model.
The module includes base logic for publishing and reading changes/events, and we've included providers for well-known data streams such as RabbitMQ and Confluent Kafka. If you use a different data stream technology, you can write your own provider or reach out to us to implement one for you.
While the setup may require technical expertise, our module includes detailed technical documentation, which can guide you through the installation and setup process. However, if you require assistance, our team of experts is available to help you get started and answer any questions you may have.
For high-volume data publishing, we recommend using our free module that integrates Data Stream Connector with OCA/queue. This module prevents your Odoo instance from slowing down, making data publishing faster and more efficient
Requirements and limitations
- Works with Odoo Community and Odoo Enterprise.
- You need to have your a data stream to use this module.
- Currently supported data streams: RabbitMQ and Confluent Kafka. To use other data streams, a provider must be developed.
- Currently it is needed to make rules configuration in code for your model. Not everything is possible from the user interface yet. See: documentation bellow.
Resources
What is Data
Streaming:
https://www.confluent.io/learn/data-streaming/
Apache Kafka:
https://kafka.apache.org/
Confluent Kafka - Enterprise-grade distribution of Apache
Kafka:
https://www.confluent.io/
RabbitMQ:
https://www.rabbitmq.com/
Integrate multiple Odoo instances
With Odoo Data Stream Connector, you are able to exchange data with other Odoo instances.
You will set-up which data models you want to send to the data stream and on the other Odoo instance, you will set-up which data you will download from your data stream. It's also possible to synchronize all data entries or only some.
Example business use cases
- One Odoo instance is source of truth for Employee Data, and you need to send it to other Odoo instances.
- Two companies that both use Odoo want to mirror some Projects between Odoo instances.
- You have a small number of employees who need Odoo Enterprise, and the majority are fine with Odoo Community. So you have two instances, but want to share some data between them.
- You're migrating from an old Odoo version to a new one, and will run your business on two Odoo instances in one year, but want to keep some data in synch.
Integrate Odoo with 3rd party applications
If you use Odoo in a larger digital ecosystem, chances are quite high, that you would like to integrate data of Odoo with either other custom made applications or off the shelf programs.
Especially in ecosystems, where many systems must exchange data, it makes sense to publish changes to data streams, so that you don't have to develop new integrations each time you need some data. You can simply connect to a data stream.
Example business use cases
- You want to exchange data between Odoo and your dedicated application written in any technology.
- You have a large fleet and you want to use a dedicated fleet software that has many more features that fleet module in Odoo. You want to download fleet data into Odoo. It's important to check with the software vendor if they support publishing changes to data streams.
Migrations
You can use Data Stream Connector also for migrations.
With one button, you can send all selected data from Odoo to a data stream and then download/consume the data in any point in time.
Example business use cases
- You want to merge two Odoo instances. You can not simply add all the data from one database to another. But you can select what data you want to export from one Odoo instance to the data stream and later download only this data.
- You want to update your Odoo to a newer version. But as you made a lot of customizations, it would be very costly to do a database upgrade. Instead, you can migrate the code, run the old Odoo and the new one simultaneously and export only the data you need to the new Odoo.
Connect to multiple data streams
Data Stream Connector is built to accommodate various data stream technologies such as Kafka, RabbitMQ, Azure Bus, and Google Cloud Pub/Sub. However, a specific provider needs to be developed to use the module with a particular data stream
Publish changes to a stream
This module publishes data stream events when a record is created or modified in your Odoo instance. You can select the data model for which changes are sent to the data stream
Delay publication of changes
We've integrated our open-source module with OCA/Queue to publish data stream changes without affecting your Odoo's performance
Data parsing
You can choose to publish raw field values or apply different parsers to each field before sending data to the data stream
Field mapping
The Data Stream Connector module offers a user-friendly interface for selecting which fields to exchange and mapping fields between data streams and Odoo
Receive data in batches
Our Data Stream Connector module enhances performance by enabling you to define batch sizes and process data in batches per data stream
Works even with relational fields
Data Stream Connector allows you to exchange not only simple fields such as char, text, and int but also relational fields. You can choose a mapping option for each relation
Send data manually
You can manually send data to a data stream, which is particularly useful when initiating the integration. With this feature, you can use Odoo's domain feature to select which data to send
WE SOLVE REAL BUSINESS PROBLEMS
We also offer
Odoo Implementation
Custom Development
Support Services
Who we are
Solvti is a software company based in Wroclaw, Poland, with customers across the whole Europe. We're a certified Odoo Partner, with projects in both Odoo Enterprise and Odoo Community.
Solvti was founded in 2021 with the mission not only to deliver software, but to solve the real business problems.
DATA STREAM CONNECTOR
Description
Base module for cross-system data exchange by using streaming. Implementation of Instances, Rules and Field Mappers.
Table of contents
Installation
- You can install all features by installing data_stream_connector_module.
- If you wish to use delay feature, make sure you have OCA Job Queue, as well as our module data_stream_queue_job installed and configured.
Usage
To use this module, you need to:
- Activate Developer Mode.
- Go to Settings and click Odoo Data Stream Connector menu.
- Create Instance with relevant provider settings.
- Add Credentials for your provider.
- Create Data Stream Rule for your model.
- Activate Scheduled Action for receiving the data or use "Manually Send Data" button for initial data sending.
Configuration
Developers
Basic Configuration
- More configuration may be required for specific provider. See more in READMEs.
- Configure all required Credentials for your provider using System Parameters or environment variables.
- Add data stream instance and select your provider.
- Add data stream rules for your instance.
Developer
- Add Data Stream Mixin ("data.stream.mixin") to a data model, for which you want to activate the exchange functionality.
class ResPartner(models.Model): _name = "res.partner" _inherit = ["res.partner", "data.stream.mixin"]
- You can use our solution for auto-generating rules using XML function and attribute in class with fields you wish to use in data streaming.
- To do so, add class attribute as list of tuples: [("field_name","fieldNameInPayload", None)]. There you can define Data Stream Rules with Mappers fields. Example use case:
_data_stream_rule_fields_send = [ ("name", "name", None), ("category_id", "category_id", None), ("company_id", "company_id", None), ("bank_ids", "bank_ids", None), ] _data_stream_rule_fields_receive = [ ("name", "name", None), ("category_id", "category_id", None), ("company_id", "company_id", None), ("bank_ids", "bank_ids", None), ]
- Use method auto_create_rules() in order to create rules automatically. You can also write your own method for that.
- Add this method to XML function to generate rules during module installation.
<function model="res.partner" name="auto_create_rules"> <value eval="[ref('base.partner_root')]" /> <value eval="True" /> <value>send</value> <value>your_module.your_data_stream_instance</value> </function> <function model="res.partner" name="auto_create_rules"> <value eval="[ref('base.partner_root')]" /> <value eval="False" /> <value>receive</value> <value>your_module.your_data_stream_instance</value> </function>
- If you do not define process for unlink, mixin will use standard Data Stream unlink behavior, which is:
_unlink_option = "restrict" # Default
- If you wish to allow users to delete records, we recommend to use soft-delete.
_unlink_option = "soft-delete"
- See more unlink option in /data_stream_base/models/data_stream_mixin.py UNLINK_OPTION_TYPE.
Send Option
You can use exchange_uid field in two ways:
- Use Odoo standard mechanism: xml_id (Default)
- Use Hashlib to create a unique Exchange UID hash.
To be able to use second option you will need to add this into your model where you added mixin:
def _sanitize_exchange_uid(self, use_hash=False): return super()._sanitize_exchange_uid(use_hash=True)
- More configuration options can be found in a particular module's README file.
Known issues / Roadmap
- two-way sync is not supported. To achieve such synchronization, you will need to create 2 rules separately. One for sending and one for receiving.
Changelog
13.0.2.0.0
- added: Implementation of Data Stream version 2.
- added: New 4 options for deleting records which were already send by data stream connector.
- added: Global auto_create_rules method to create rules through xml files.
- changed: Improved sending mechanism.
- changed: Improved receiving mechanism.
- changed: active field is not longer required in data stream connector.
- changed: False value in send mechanism will be present only for Boolean fields.
- fixed: relations search mechanism.
- removed: Additional Rules.
13.0.1.0.2
- changed: Updated README, Description
- changed: Refactors
13.0.1.0.1
- added: Possibility to use delay feature.
13.0.1.0.0
- Initial implementation of Data Stream Connector modules.
Odoo Proprietary License v1.0 This software and associated files (the "Software") may only be used (executed, modified, executed after modifications) if you have purchased a valid license from the authors, typically via Odoo Apps, or if you have received a written agreement from the authors of the Software (see the COPYRIGHT file). You may develop Odoo modules that use the Software as a library (typically by depending on it, importing it and using its resources), but without copying any source code or material from the Software. You may distribute those modules under the license of your choice, provided that this license is compatible with the terms of the Odoo Proprietary License (For example: LGPL, MIT, or proprietary licenses similar to this one). It is forbidden to publish, distribute, sublicense, or sell copies of the Software or modified copies of the Software. The above copyright notice and this permission notice must be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Please log in to comment on this module