A Real-Time Data Stream Processing Model for a Smart City Application Leveraging Intelligent Internet of Things (IOT) Concepts.
ABSTRACT
Due to the vast amount of data that is being generated by the sensors through the smart devices in smart cities, streams of data must be processed in real time to gain insight quickly and to make decisions that are in most cases critical and time sensitive.
The difficulty is diminished by using big data methods such as Cassandra, Hadoop, Kafka and Spark to perform real-time stream processing in an Internet of Things (IoT) environment, such as traffic monitoring in a smart city environment.
Among the different dimensions that improve the quality of life of people in a smart city, one of the very important one is transportation.
Intelligent Traffic Monitoring System (ITMS) in a smart city, monitors traffic by detecting and displaying what is occurring on a particular road.
In this thesis, a real-time data stream processing model was developed and used data streaming trends to monitor traffic in an ITMS.
TABLE OF CONTENTS
CERTIFICATION ….. ii
ABSTRACT ……. v
ACKNOWLEDGEMENT …………….. vi
DEDICATION …. vii
LIST OF FIGURES ………… xi
CHAPTER ONE INTRODUCTION … 1
1.1 Problem Statement …. 5
1.2 Objectives …………………………………………. 6
1.3 Thesis Organization …………………………………………… 6
CHAPTER TWO LITERATURE REVIEW …………… 7
2.1 Smart City Concepts …………………….. 7
2.1.1 Smart Mobility ……………………… 7
2.1.2 Smart Grid …………………………… 8
2.1.3 Smart Buildings……………. 8
2.1.4 Smart Water ……………………………. 9
2.1.5 Smart goods ………………………. 10
2.1.6 Smart Industry ……………….. 10
2.1.7 Smart Lightning…………….. 10
2.1.8 Smart Waste Management …………….. 10
2.1.9 Intelligent Traffic Monitoring Systems …………. 10
2.1.10 Smart Energy Management ……………… 11
2.2 Architecture of Smart Cities ……………… 11
2.2.1 Urban Area …………. 11
2.2.2 Dense and heterogeneous devices …. 12
2.2.3 Types of Data …… 12
2.2.4 Communication Techniques………. 12
2.2.5 Control Centre Server ………… 12
2.3 IoT for Smart Cities ……………………. 13
2.4 IoT Architecture ……………… 14
2.4.1 Message Queue/Stream processing block ……………. 15
2.4.2 The Database Block …………….. 15
2.4.3 The Distributed File System Block ………. 15
2.5 Challenges of IoT …………. 15
2.6 Traffic management system ……………………………. 16
2.6.1 Smart Parking System …………………… 16
2.6.2 Smart Street Lights ………………….. 17
2.6.3 Public Transport ………………………… 17
2.7 Real-Time Data stream processing model …….. 18
2.8 Related works …………………………………………………. 19
2.8.1 Cost effective road traffic predictive model using Apache Spark …………………….. 19
2.8.2 Advanced traffic management system using IoT ……………………. 22
2.8.3 Smart traffic light in terms of the Cognitive Road Traffic Management System (CTMS) based on the IoT 24
2.8.4 Traffic accident analysis using neural networks and decision trees …………………. 25
2.8.5 Big Data Analytics Architecture for Real-Time Traffic Control ………………………… 26
CHAPTER THREE METHODOLOGY
3.1 Apache Kafka …………………………………………………………………………………. 27
3.2 Apache Spark ………………………………………………………………………………… 28
3.3 Spark Streaming …………………………………………………………………………….. 28
3.4 Real-time Integration of Apache Kafka with Apache Spark …………………….. 29
3.5 Cassandra ……………………………………………. 29
3.6 Spring boot …………………………………………………………. 30
3.7 Architecture of the proposed system ………………………………………………….. 30
3.8 The Producers ……………………………………………………………………………….. 31
3.9 The Consumers………………………………………………………………………………. 32
3.10 The results …………………………………………………… 32
3.11 Java …………………………… 32
CHAPTER FOUR IMPLEMENTATION AND RESULTS
4.1 Introduction ………………………………………….. 33
4.2 Apache Zookeeper ………………………………………………………. 33
4.3 Kafka and Zookeeper servers ………………………………………………… 34
4.4 Spark and Cassandra ……………………………………………….. 36
4.5 Kafka producer ………………………………………………………. 37
4.6 Spark Streaming ……………………………………………………… 37
4.7 Streaming statistics ………………………………………………. 40
4.8 CHALLENGES ………………………………………………………………. 41
CHAPTER FIVE SUMMARY, CONCLUSION AND RECOMMENDATION
5.1 Summary……………………………………………………………………. 42
5.2 Conclusion ………………………………………………….. 42
5.3 Future work …………………………… 43
REFERENCES 45
INTRODUCTION
Quintillion bytes of data are being generated daily and handling this enormous amount of data is becoming more tedious every day.
These bytes of data are generated by people using devices such as mobile phones, laptops, smart devices and these devices are connected to the internet so as to be able to identify themselves to other devices.
These devices are found everywhere in a smart city (Gehlot, 2016). According to (Santana, Chaves, Gerosa, Kon & Milojicic, 2016), a Smart City is a city in which social, business, communication, and technological aspects are supported by Information and Communication Technologies such as intelligent IOT and data collection sensors to improve the experience of the citizen within the city.
To achieve that, the city provides public and private services that operate in an integrated and sustainable way. The bytes of data generated by the sensors must be used to make data – driven decisions as they are generated in real time proactively.
As the sensors sense environments continuously, data streams generated must be processed in real- time to gain insight quickly because the data generated are in many cases critical and time sensitive. Smart cities are built on the Internet of Things.
REFERENCES
Al Nuaimi, E., Al Neyadi, H., Mohamed, N., & Al-Jaroodi, J. (2015). Applications of big data to smart cities. Journal of Internet Services and Applications, 6(1), 25. https://doi.org/10.1186/s13174-015-0041-5
Amini, S., & Prehofer, C. (n.d.). Big Data Analytics Architecture for Real-Time Traffic Control, (Tum Llcm).
Apache Kafka. (n.d.). Architecture of smart cities. (n.d.).
Caliri, G. V. (n.d.). Introduction to Analytical Modeling.
Chong, M. M., Abraham, A., & Paprzycki, M. (1997). Traffic accident analysis using.
Cities, S. (2015). IoT What is IoT? Corporation, I. B. M. (2013). Hive © 2013.
Gehlot, R. (2016). Storage and Retrieval of Data for Smart City using Hadoop, 3(5), 85–89.