Trace Connect

Background

At the end of 2022, we released our mobile application to the community. Open sourcing is a fundamental aspect of our vision, and there are evident benefits associated with it. We have been developing the Trace app since the second half of 2021 and have tested it on the field for over two years with two farmer cooperatives in Indonesia.

The application is designed to cater to collectors who traverse remote farmlands to acquire harvested goods. Our primary aim was to ensure the app's simplicity. It is developed using React and primarily serves two key functions: enrolling farmers and recording their on-site transactions. When enrolling farmers, collectors attach an NFC/QR code-based card, which we provided them in advance, and share it with the farmers. These cards serve as a means for farmers to validate transactions when delivering their products and confirming payments. In essence, the app streamlines the collection process and facilitates farmer-verified payments.
Additionally, the app offers support for capturing sales receipt photos in case a farmer forgets their card. It operates seamlessly both online and offline. When connected to the internet, the app syncs with Trace, allowing users to access and download all transactions online by logging into Trace.

The app was highly appreciated by cooperatives, and the farmers were enthusiastic about the cards. Since the launch, it has effectively digitised close to 15,000 Nutmeg transactions from rural Indonesian islands. To enhance fairness and transparency, we introduced a digital passbook that is connected to the card. This passbook allows farmers to conveniently access and review their past transactions and payments online by using their card numbers.

We are very much aware open source app without the backend is kind of useless. But we already knew open-sourcing the complex Trace backend is not the ideal solution if you really want people to use it. So we decided to create a small portion of Trace which only covers the basics required to run the app.

As we contemplated the idea, we came to the realisation that we could employ the same system to establish connections between external systems and data sources with Trace. This would essentially transform it into a central hub for both the inflow and outflow of data within the Trace ecosystem.

Goals and Objectives

Our goal with Trace Connect is to achieve the following objectives:

Standalone Backend for Trace Mobile App: Serve as a backend for the Trace mobile app, allowing users to deploy their own cloud-based instances of the app independently of the main Trace system. This will include configurations like premium payments and access control.
Data Synchronisation: Guarantee smooth data synchronisation between the Trace mobile app, Trace system, and other external systems, enabling effortless two-way data transfer. Enable the easy import of data from external sources into Trace, as well as the efficient export of data from Trace to external systems, to ensure a fluid exchange of data throughout the ecosystem.
Integration Hub: Act as a central integration hub, facilitating the exchange of data between external systems and ERPs with Trace through APIs and the Connect SDK.
Additional Data Collection Agent: Enable the Trace app to support the creation of new or extra fields for data-gathering purposes. While Connect will function as a data collection agent, focusing mainly on basic farmer data and transactions, it is not intended for use as a survey instrument. However, it can serve as a central point for integrating additional data from various systems.
Modular and Scalable: Design the system to be modular and scalable, allowing for future expansion and integration with evolving technologies and requirements.

System Architecture

The TraceConnect server architecture comprises a Django-based server application that serves as the central hub for seamless data processing and management. It integrates with both PostgreSQL for structured data and MongoDB for unstructured data storage. Through APIs, it establishes connections with the Trace mobile app and external applications, implementing OAuth for external applications and JWT for the Trace mobile app's secure authentication. Furthermore, it features a user interface for manual data export, offering users flexibility and control over data management processes.

Trace communicates with the Connect using a shared SDK, which also serves as a bridge to connect with external ERP systems. When generating OAuth credentials, users have the flexibility to configure different scopes according to their specific requirements.

Core Modules

The core modules of TraceConnect primarily revolve around companies, farmers, their transactions, and the customization of data connections for the Trace mobile app. Here are the key core modules:

Authentication (Auth): This pivotal module is responsible for user management and access control within TraceConnect. It empowers administrators to define and manage user roles and permissions, ensuring that users have the appropriate level of access to the system's features. In addition to handling internal user authentication, the Auth module plays a crucial role in managing OAuth connections for external services. This includes the secure integration of external applications and services, enhancing the overall security and functionality of TraceConnect.
Supply Chain: The Supply Chain module serves as the backbone of the system, encompassing the comprehensive management of the supply chain structure. This includes the tracking of companies, farmers, products, and their interconnected relationships. Within this module, the crucial task of managing farmer cards is executed, providing a seamless means of validation during transactions and interactions within the supply chain. The Supply Chain module ensures that all stakeholders are effectively interconnected, contributing to the streamlined flow of goods and information.
Transactions: At the heart of TraceConnect's functionality lies the Transactions module, where all transactions between companies and farmers are meticulously recorded and managed. This encompasses a wide range of activities, including product transactions and the tracking of farmer payments.
Configurations (Configs): The Configurations module is dedicated to tailoring the TraceConnect experience to suit specific requirements. It offers administrators the ability to customize various aspects of the app's functionality. This includes configuring premium settings, allowing for the fine-tuning of features, and controlling user access privileges. Importantly, administrators can make on-the-fly adjustments, such as enabling or disabling the "Buy" feature for specific collectors, ensuring that the system remains agile and adaptable to evolving needs.
Forms: Within the Forms module, administrators can configure custom forms to collect additional data relevant to their specific workflows. These forms provide flexibility and versatility, allowing TraceConnect to capture and process a wide array of data types. Whether it's capturing unique data points or adapting to changing data requirements, this module empowers users to enhance their data-capturing capabilities, ultimately improving the precision and completeness of their records.

Database Architecture

Our database architecture has been thoughtfully designed to prioritise simplicity and minimalism while maintaining efficient data management. Central to this architecture is the "company" table, which serves as the core entity around which all data structures revolve. This approach ensures clean separation and allows multiple companies to utilise the same server without data overlap.

In the provided high-level database architecture diagram, the following key elements are illustrated:

Company-Centric Structure: Each company has the autonomy to onboard its own collectors as app users, thereby establishing a dedicated user base within the system.
Customisable Preferences: Companies have the flexibility to define their app preferences, such as specifying premium requirements, configuring app functionality, and setting their preferred products for collection.
External Connections: Companies can tailor their external connections and define the scope of their interactions within the system, granting them control over data flow and integration.
Farmers: All farmers are onboarded or connected to specific companies, fostering a well-organised and efficient ecosystem.
Product Transactions: The flow of product transactions occurs from farmers to the company table, creating a clear linkage between the products collected and the associated payment transactions.
Payment Transactions: Payment transactions, vital for compensation to farmers, flow from the company to the farmer. The verification process is conducted seamlessly through the use of farmer cards.

Operational Workflow

In the preceding section, we elaborated on how Trace Connect can effectively operate as a data collection server, highlighting the necessary configuration steps for seamless functionality. To illustrate the simplicity of these steps, let's dive into the operational flow. For this demonstration, we assume that the Trace Connect server is already up and running.

Company Setup: To establish your presence within the system, the initial step involves creating your company account. As elucidated in the earlier sections, the entire data structure centres around the concept of 'companies,' forming the core of your data ecosystem.
Configurations: Once your company account is in place, you can proceed with essential configurations. This includes adding the specific products you intend to collect using the application. Additionally, you can designate and onboard collectors or users under your company who will utilize the app. Within the app's configuration section, you gain the flexibility to define various parameters. Customise your preferences, such as setting premiums for different product types and enabling distinct functions like buying, selling, and processing farmer payments.
Custom Forms: Tailoring your data collection process is made possible through the custom forms section. Here, you can establish additional data fields and seamlessly link them to existing tables such as 'farmer' and 'transactions.' These newly configured fields will seamlessly integrate into the relevant sections of the mobile app after the next synchronisation. This facilitates efficient data collection in the field, data validation, and data cleansing, particularly when interfacing with external systems. All custom forms are stored in JSON format within the Connect database.
Connect External Systems: The capability to interface with external systems is a valuable asset, whether you aim to export data collected via the app or import data from external sources. To establish these connections, you have the ability to create credentials, specifying the scope based on your specific use case. For instance, you can configure credentials for reading and writing data related to farmers. This functionality extends the utility of the Connect server as an integration agent, allowing you to incorporate multiple data sources and seamlessly channel data into one or more other systems.
App Connection: The process of connecting the mobile app is straightforward. Simply install the mobile application from the Google Play Store, input your server URL, and log in using your collector credentials. The application will seamlessly synchronise with your server, ensuring that all relevant data is accessible within the app, thereby facilitating efficient data management and collection.

Data Fields

In the context of data gathering, the Connect server is primarily centred on collecting information related to farmers and their transactions. The data fields within this category are divided into two main sections. The first section comprises 'Base Data Fields', which include default fields such as name, address, etc., that are automatically collected in Connect. The second section consists of 'Additional Fields', which are customisable and can be tailored according to specific requirements.

Base farmer fileds

Field	Type	Required	Description
first_name	char	Yes	First name of the Farmer
last_name	char	Yes	Last name of the farmer
street	char	No	Street name
city	char	yes	City or village
country	char	yes	Country
province	char	Yes	Province or State
post_code	char	No	Postcode
latitude	float	No	Latitude of the farm location
longitude	float	No	Longitude of the farm location
country_code	char	No	Phone Country code
phone	char	No	Phone or mobile number
email	char	No	Email address
gender	char	No	Gender of the farmer
dob	date	No	Date of birth
house_hold_size	int	No	Number of family members
description_short	char	No	Short description of farmer
description_long	char	No	Long description of farmer

Base Transaction Fields

Field	Type	Required	Description
source_id	char	Yes	Transaction source name or code. In the case of farmer transactions, this can be a global gap number.
destination_id	char	Yes	Transaction destination name or code
product_code	char	Yes	Unique product name or code
quantity	float	Yes	Total transaction quantity
unit	char	Yes	Product unit
price	float	No	The total price paid to the farmer
currency	char	Yes	ISO currency code
date	date	Yes	Transaction date
invoice	file	No	Invoice number
buyer_ref_no	char	No	The number used by the buyer to identify the stock
seller_ref_no	char	No	The number used by the seller to refer to the stock

Additional fields

Companies have the ability to define extra fields related to farmers and transactions on the Connect server. Once these additional fields are set up on the server, they will be incorporated into the app during the subsequent synchronization process. For example, a field named 'Living Income' could be added to the farmer form. Consequently, the next time a collector registers a new farmer, this field will appear in the form. This feature is also beneficial when integrating data from external systems. By specifying the field and its type, the system ensures proper validation.

At present, we accommodate basic data types in these custom fields. However, there is always the potential to broaden this range to cater to different use cases.

Character filed
Number
Date
Latitude
Longitude

APIs

We are still working on publishing our APIs and the detailed documentation. We have implemented the following APIs.

Name	Methode	Description
List Farmers	GET	Retrieve a list of all farmers.
Create farmer	POST	Create a new farmer.
Update farmer	PUT	Update an existing farmer's information.
Delete farmer	DELETE	Delete a farmer.
List transaction	GET	Retrieve a list of all transactions.
Create transaction	POST	Create a new transaction.
Attach transaction invoice	PUT	Attach an invoice to a transaction.
List payments	GET	Retrieve a list of all payments.
Create payments	POST	Create a new payment.
Attach payment invoice	PUT	Attach an invoice to a payment.
List premiums	GET	Retrieve a list of all premiums.
List products	GET	Retrieve a list of all products.
Attach farmer card	POST	Attach a card to a farmer.

Roadmap

November 2023: initial deployment and mobile app linking
Jan 2024: publishing the APIs and first open-source release
Feb 2024: Publishing APIs and external server connection
April 2024: enable Trace SSO
June 2024: First Python SDK release