Our overall approach and architecture of the Freight Pricing Engine, which has been conceptualized to make the process of price discovery in the trucking marketplace hassle-free and transparent.
‘RIVIGO Freight’ is a data and technology driven freight marketplace that aims to digitize trucking, making it exceedingly simple with everything available at a single click.
Pricing is one of the biggest enablers for the success of any marketplace. Yet there is negligible price transparency in the trucking ecosystem today. In addition to this, the process of price discovery and finalization is also very tedious for the customer given the low-level of technology adoption and complexities within the sector.
Frieght Rate Exchange was conceptualized to make the process of price discovery hassle-free and transparent. We will be covering this concept through a series of articles. In this first article, we will introduce our overall approach and specifically cover the architecture of the Freight Pricing Engine.
In order to understand this series well, it would be good to know the basic terminology in the logistics marketplace sector. Some such important terms are:
Further, there are multiple factors that affect the pricing of trucks. The dependence on such variables makes this a pretty difficult problem to crack. Some of these factors are:
When we started our work on this project, we had a static list of N source and destination pairs with price points for all truck types. This translated to over 1 million data points.
Most of this data was created using basic market research and extrapolation logic in between lanes. A web and mobile interface for the business team enabled them to report prices for any ODVT real-time. Basis certain empirical formulae established through market research, every newly collected price was used to predict prices for related ODVTs. These formulae were codified into multiple heuristics-based rule books to make the system update prices real-time basis any new input.
Once data liquidity and user footfall on the platform increased, a system was created to auto-update lane prices. This was done using historical price data and corresponding user traffic on our platform to determine market velocity (demand-supply equilibrium state). This system is named after the great Indian mathematician, Srinivasa Ramanujan. His achievements in Mathematics inspire the work on our pricing engine.
The Ramanujan System uses machine learning to train multiple models to predict prices on similar lanes. It uses various factors and data points captured from our application and historical data to do so. These models compete amongst each other basis the ‘Reliability Engine’ scores and result in dynamically updated prices on the system.
We will share more details about the Ramanujan System and its applications in our upcoming posts.
The diagram below shows how data flows in between various components of pricing engine.
The description of the stages below will explain how we calculate prices on lanes.
User activities are critical data points for us. These include things like how much time they spend on monitoring the price on the platform, how many times they quote on a particular ODVT etc. These user activities are captured via the Event Handler system.
Event Handler system pushes these events to SNS (Simple Notification Service), where queues (SQS – Simple Queuing Service) are subscribed. We have a daemon worker running which stores the events from the queue in the Mongo Database cluster.
Other events like load posted, load accepted etc. are captured by backend systems (supply and demand) and saved directly in our MySQL Database clusters.
These database sources are used by our pricing engine as input sources.
Price Update Collecting Worker
Price update collecting worker reads directly from Queues (SQS) which is subscribed to different SNS Topics. This is deployed in an AWS EC2 instance. The main purpose of update collecting worker is to:
We used Mongo database cluster because it provides:
Besides using Mongo database clusters, we also use a PostgreSQL database cluster for storing lane and location information. We have stored lane information in PostgreSQL because of PostGIS, which enables fast querying on geospatial data.
Update collecting worker also adds data related to lane information using (src lat, src lon, dst lat, dst lon) combination.
Derived Price Updates Generating Worker
Derived price updates generating worker is the system which generates price updates for linked ODVTs from a single ODVT update. This is deployed in an AWS EC2 instance.
This worker reads updates from the mongo cluster, creates new lanes, calculates reliability of the updates, derives related price updates and saves price snapshots in AWS S3 at regular intervals, which are fetched at booting time for speeding up the initialization of the worker. Data related to the recent price points and user reliability is stored in Redis. This enables real-time reliability calculations.
Pricing Engine API
Pricing Engine API is the server which backend services (supply and demand backend) hit to get prices of a particular ODVT. We have hosted this service in AWS Elastic Beanstalk. Load balancers are used to balance loads in the servers.
Pricing Engine API connects with the following systems:
We measure the accuracy of our pricing engine by measuring pricing offset per trip. This is defined as the percentage difference in the sourcing price versus predicted price. In the graph below, one can see the clear trendline of decreasing pricing offset per trip. There has been close to 10x improvement.
Certain peaks are visible intermittently. This is because of multiple factors or externalities like quarter end, flood in a city etc., which the pricing engine has to further improve upon.
Freight Pricing Engine has been able to bring in much-needed price transparency in the sector. This offering has led to an exponential increase in the number of trips, making RIVIGO Freight the leading trucking marketplace in India in a very short span of time. However, we believe this is just the beginning and we have a long way to go.
In the subsequent articles in this series, we will talk about:
We are continuously innovating and improving engine performance through machine learning as data liquidity continues to improve, to revolutionize the USD 100 Bn Indian trucking marketplace.
More stories by Tushar Makkar
This article gives an overview of how a hackathon team at RIVIGO made AI based Conversational CRM Agents, to enable hyper-growth and infinite scalability in connecting with customers while reducing manual interventions. Introduction Rivigo Freight is a technology-enabled logistics marketplace that connects shippers to carriers. Around a hundred thousand shippers and two lakh carriers