Freight Rate Exchange was conceptualized to make the process of price discovery in the trucking marketplace hassle-free and transparent. In Part 2 of this series, we dive into the architecture design of this engine.
In the first part of this series, we had introduced Freight Rate Exchange (FRE) and the objectives for the same. FRE enables real-time freight information for 1 Mn+ lanes based on ODVT (origin-destination-vehicle type). This was envisioned to bring price transparency to the otherwise unorganized freight market.
Every shipper/carrier operates in different locations and lanes with various vehicle types. Therefore, it was important that we showed relevant personalized ODVT feed for every user. Enabling this would create real value for users and improve user stickiness on our network.
When a user visits FRE, we capture his/her actions. These actions could be lanes viewed, time spent on cards, price quotes etc. This information is then used as features in multiple AI engines to improve the system accuracy, in a self-learning manner.
Following were the basic requirements for architecture design.
To solve for these requirements, we utilized the micro-services architecture and delegated tasks to appropriate services.
For the price feed for any user (shipper/carrier), we needed to ensure that the recommended lanes for prices were relevant and their pricing was accurate. This meant that we show top lanes with the most recent prices. At the same time, we needed to show the price seamlessly without any delays / latency.
To ensure the price feed is always available, we persisted a pre-processed collection of the entire lane set with its address details in ten regional languages in Mongo DB. For customized user feed, we used Redis to store a set of routes for each user (using user route recommendation engine) as a cache that gets invalidated every few minutes.
ODVT price search
The user route recommendation engine is also a learning system and the accuracy of the feed is linked to the extent of user activity on product. Therefore, we decided to allow users to search price for any route. We used AWS-hosted CloudSearch service to enable search functionality. It stores information regarding all the lanes, provides location-based indexing and sorting of indexes on huge amounts of data efficiently.
Given that the reliability of the pricing engine was gradually improving, it was important to enable crowd sourcing of prices as well. It served the following three purposes.
Therefore, users were given the option to enter the prices on lanes. We delivered these inputs to our route recommendation engine using hosted services like AWS SQS and SNS. The input prices were used to dynamically update pricing engine prices within permissible variance norms to visually deliver impact of user’s price input event. This enabled better user engagement.
For every user, recommended ODVTs are generated by user profile creation. User profiles are created using user activity events like search, order clicks, loads view, load posts, price view etc. This will be covered in more detail in a future blog on recommendation engine.
For day 1 users where zero user activity exists, initial feed logic was defined as given below.
Post initial feed, update of relevant ODVTs happen on real time basis using data from event handler. This is done using a caching mechanism for the profiles which help in making it a high throughput engine.
Our lane universe is not static and expands with every new order / user search event. This means that we need to inform consumer services (downstream services) regarding the newly added lanes regularly. This is done using AWS SNS subscriber publisher model. FRE consumes this information and updates relevant ODVT information.
Moreover, whenever a user quotes price and a price advantage exists for the user on platform, notifications are sent in real time to the user. Given the dynamic nature of prices, we cache ODVTs for which prices have been quoted and data is made available using event driven architecture.
Pricing engine receives price inputs on an ODVT from backend infrastructure framework. These inputs are ingested using SQS and processed by derivative worker which generates derived updates. To know more about the architecture of the derivative worker please refer this blog.
On receiving price inputs from users, a reliability score for the quote is calculated. This is done by using machine learning models. More details about the model will be shared in subsequent blog posts.
Once the reliability of a particular user’s quote is determined, the next step is to decide whether to consider the quote’s impact on price or not. Entire time series of price inputs with respective reliabilities and recency-based weightage function is used to determine the new price point. The code snippet below demonstrates the exponential delay function in finding reliability of quotes.
Using this price, we create and modify price cluster spread for each ODVT and correspondingly update the price as the centre of the applicable cluster.
At RIVIGO, we solve real world problems and therefore, the delivered impact of any work is the most important yardstick to measure technology. We were able to provide a seamless and hassle-free experience to users by delivering price information for routes swiftly and precisely. Traffic on FRE has been on a constant rise proving that price discovery is a real challenge for users and hence, a solution like this was much needed. The number of FRE quotes received has also been on a constant rise. We received valuable inputs that improved accuracy of our recommendation and pricing engines.
In the next article, we will cover the approach and architecture of the Ramanujan Price Prediction Engine, which uses machine learning to predict demand supply velocity for every ODVT and train multiple models to update prices without any manual inputs.