Illegal dumping is an issue for cities, who see their street flooded by littered despite their attempts to remove it.
Antwerp is determined to act against fly tipping in an new way. Thanks to the experience gained during the WINPOL project and with the support of the Interreg Europe programme, the Belgium city is working on the development of an highly innovative system to fight illegal dumping using an app installed in a smartphone.
A simple question started it all: can we detect litter on the street so that a smartphone app could be built to take pictures, recognize the litter, categorise it and dispatch it with its geocoordinates to the right team to collect it? And cherry on the cake, could this be done using only open-source tools? If the tools were easily found, getting the right data was a huge barrier.
How to build an artificial intelligence library on litter?
A lot of artificial intelligence libraries are available to perform image recognition, but none of them are specifically built to recognise litter. This means that the City of Antwerp had to train these machine learning models from scratch. Moreover, source images of litter are not abundantly present on the internet. So, an alternative way had to be found to provide sufficient training data.
The Digipolis (Digital Antwerpen) team started to collect a minimum set of a thousand images per category (e.g. sofas, chairs, television sets, computer screens, standard and non-standards garbage bags, small shopping bags, etc...) and dug for images in the city’s litter notification portal. Then, it was necessary to label the objects visible in these images: what is visible on each image and where is it located? Bounding boxes were developed, and the objects started to be labelled and fed the data of the machine learning model being used.
For some categories, the recognition process was taking off well, but for most of them, it was simply not enough data to obtain good recognition. More data was needed. Thus, a group of students worked on labelling much more data (thousands of extra images), now segmenting the object from its background.
A long way to image recognition
Results improved, but they were still far from a successful working app that reaches its goals. They also saw that there was some overfitting on backgrounds, colours and other things that were not related to the litter itself. We needed another approach. Improvements continued.
The Digipolis team set up a system to augment the training data: rotation of the pictures, colour changes, skewing, flipping, noise, etc... Slight improvements in the results were shown, but still, not sufficiently close to the target.
In a last attempt, it was decided to take again the labelled images and built a piece of software to remove all litter objects from the source data, filling up the gap with similar background from the same image and put another (arbitrary) litter object from the data set.
Following this move, the data set was much richer, and the newly trained models displayed better results (confusion matrix with a clear diagonal).
13 litter categories in the radar
The long hours of work paid off, and Antwerp is now running its app. The model put in production is recognising 13 litter categories and the app is currently in use within the city for evaluation. The city even expects to receive demands to extend the model to some 30 categories. The improvement continues!
This action, entitled "Image Recognition for reporting of fly-tipping/illegal dumping" is part of the plan developed by the City of Antwerp to improve waste management performances on its territory by fostering the use of intelligent equipment and planning derived of it. This Action Plan implemented by Antwerp includes two other changes to its Strategic Multi-Annual Plan 2020-2025. Discover them here.
