Lambda Architecture

Home / Evangelists / Lambda Architecture

In the Lambda Architecture website we have a brief history and description of the architecture. “Nathan Marz came up with the term Lambda Architecture (LA) for generic, scalable and fault-tolerant data processing architecture, based on his experience working on distributed data processing systems at Backtype and Twitter.
The LA aims to satisfy the needs for a robust system that is fault-tolerant, both against hardware failures and human mistakes, being able to serve a wide range of workloads and use cases, and in which low-latency reads and updates are required. The resulting system should be linearly scalable, and it should scale out rather than up” [1].

 

Fig.1. Lambda Architecture (from http://lambda-architecture.net/)

 

In the site is also described how it works:

  1.  All data entering the system is dispatched to both the batch layer and the speed layer for processing.
  2. The batch layer has two functions: (i) managing the master data set (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views.
  3. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way.
  4. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only.
  5. Any incoming query can be answered by merging results from batch views and real-time views. [1]

The Lambda Architecture is for applications that need low latency (from few seconds to a few hours). A news recommendation system that uses various sources, processes and normalises the input, and then indexes, ranks and stores it, is a good example of an application of this architecture [2].

References

  1. Lambda Architecture, http://lambda-architecture.net/
  2. Questioning the Lambda Architecture , https://www.oreilly.com/ideas/questioning-the-lambda-architecture
Ricardo Santos
Integrating the world for over 10 years and enthusiastic about the Internet of Things. I help to spread the word at Polarising about the future that is happening today. Martial artist and History nerd, hoping technology will help us get where we need to go.
Recommended Posts
  • IOT Middleware
    IOT Middleware
    There are several ways to process and integrate data but due the lack of standards and the heterogeneity of the “things”, a key role is played by the middleware. Middleware is application-independent software that provides services that allow communications between applications. Middleware hides the complexities of the lower layers, like operating system and network, in order […]
  • Edge Computing
    Edge Computing
    Edge computing takes localised processing a bit further than Fog Computing, because it allows for actions to be taken on-site, in the processing point. This poses an advantage over Fog Computing as there are less points of failure. Each item in the chain is more independent and capable of determining what information should be stored […]
  • Fog Computing
    Fog Computing
    Even though Cloud computing is a great way of processing the data generated by the “things”, it doesn’t meet all IoT’s needs. For instance, one issue that affects the quality of service (QoS) severely is network latency. Real time applications are affected by the delay caused by latency in networks [1]. For example, when the […]
  • AWSome Day @Lisbon, the Amazon Web Services Event
    AWSome Day @Lisbon, the Amazon Web Services Event
    Last May 18th, AWS (Amazon Web Services), pioneer and leading provider of cloud services, held the AWSome Day event in Lisbon, a full day tour of its services. Since Polarising is keeping a close watch on the cloud space, I was very interested to hear what they had to say and the maturity of AWS’ […]

Leave a Comment