Journal of Computer Technology and Software

Vol. 1 No. 2 (2022)
Articles

The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction

  • Alkis Samaras

Published 2022-04-30

How to Cite

Samaras, A. (2022). The Log: What Every Software Engineer Should Know About Real-Time Data’s Unifying Abstraction. Journal of Computer Technology and Software, 1(2). Retrieved from https://ashpress.org/index.php/jcts/article/view/337
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Applications increasingly need to process data as continuous streams rather than as finite datasets. This paper argues that an append-only log provides a fundamental abstraction for building distributed systems. By using logs as the central data structure, systems can simplify replication, fault tolerance, stream processing, data integration, and event sourcing. The paper discusses how technologies such as Apache Kafka leverage the log abstraction to build scalable and reliable backend infrastructures for modern applications.

References

  1. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., et al. (2007). Dynamo: Amazon's highly available key-value store. Proceedings of the 21st ACM SIGOPS Symposium on Operating Systems Principles (SOSP '07), 205–220. https://doi.org/10.1145/1294261.1294281
  2. Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., et al. (2008). Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2), Article 4. https://doi.org/10.1145/1365815.1365816
  3. Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google File System. Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP '03), 29–43. https://doi.org/10.1145/945445.945450
  4. Dean, J., & Ghemawat, S. (2004). MapReduce: Simplified data processing on large clusters. Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), 137–150.
  5. Lakshman, A., & Malik, P. (2010). Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2), 35–40. https://doi.org/10.1145/1773912.1773922
  6. Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., et al. (2013). Spanner: Google's globally distributed database. ACM Transactions on Computer Systems, 31(3), Article 8. https://doi.org/10.1145/2491245
  7. Ongaro, D., & Ousterhout, J. (2014). In Search of an Understandable Consensus Algorithm (Raft). Proceedings of the USENIX Annual Technical Conference (ATC), 305–319.
  8. Gilbert, S., & Lynch, N. (2002). Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601
  9. Vogels, W. (2009). Eventually Consistent. Communications of the ACM, 52(1), 40–44. https://doi.org/10.1145/1435417.1435432
  10. Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 1–7.
  11. Barlocker, B., & Yan, X. (2021). Contrastive Representation Learning for Anomaly Detection in Cloud-Based Backend Services. Artificial Intelligence and Computing Innovations, 1(2).