Bottle counting system for a re-cycling facility based on processing of depth images

Abstract

Keywords: Kinect; Depth İmage Processing; Camera Calibration; Bottle Counting

Introduction

Although Kinect was initially invented for the game industry in 2010, it has been utilized widely for image processing applications owing to its low consumer price, compact size, and capability to capture depth image data [1-2]. Depth images could be obtained by either high cost 3D cameras or stereovision before the launch of Microsoft

Kinect for Windows sensor v1. Kinect-v1 sensor measurements are susceptible to the disturbances since it adopts structured-light method for depth measurements. In order to improve the depth measurement accuracy, Microsoft has released Kinect v2, which has a time of flight (tof) depth camera. The captured depth images from Kinect v2 have

*Corresponding author. *E-mail address: gurkan.kucukyildiz@usak.edu.tr This paper was recommended for publication in revised form by Regional Editor N. Özlem Ünverdi Published by Yıldız Technical University Press, İstanbul, Turkey Copyright 2021, Yıldız Technical University. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

better quality compared to Kinect v1. On the other hand, researchers still focus on the improving Kinect depth measurement accuracy [3]. Kinect is utilized in various applications in the literature. Suat et. al. have developed an image processing based volume measurement system using Kinect sensor. The authors proved that the developed system has higher measurement accuracy than traditional point laser based systems [4]. Atıf et. al. utilized the Kinect sensor for underwater 3D Scene reconstruction [5]. Kinect is covered with a developed waterproof housing such this purpose. The authors proposed a time of flight correction method in order to overcome refraction of light problem due to housing and water. The authors pointed that the developed system is suitable for coral reef mapping and underwater SLAM. Neto et. al. have developed a Kinect based face recognition system [6]. The authors proposed an algorithm based the variation of the K-nearest neighbors’ algorithm over histogram of oriented gradient descriptors dimensionally reduced by principal component analysis. Authors claimed that the developed algorithm outperforms traditional face recognition methods while requiring much less computational resources (memory, processing power and battery life) when compared to existing techniques in the literature. Sun et. al. proposed a novel facial descriptor by processing Kinect depth frames [7]. The authors developed a tensor representation for histogram arrays embedded in multidimensional space and further use multilinear principal component analysis. To evaluate the developed method performance, the authors have applied the developed model on to the two different public databases constructed with Kinect depth camera: CurtinFaces and Eurecom. The authors pointed that the developed algorithm have over than %86 classification accuracy for both databases. Vera et. al. have proposed a pedestrian counting system using array of Kinect sensors placed in zenithal position [8]. Once the developed algorithm separately detects the person by the Kinect array, the algorithm then constructs the tracklets of the detected persons’ based on their closeness and time stamp. The authors claimed that the developed method is practical solution to the related problem owing to the low computational cost. Yan et.al. have developed a depth image based anthropometric clothing measurements from 3D body scans [9]. The authors could determine the body model using a non-rigid ICP to fit a pre-defined model. Such this purpose, nonlinear regression based anthropometric measurement estimation was applied. The author have applied their developed algorithm to the a public benchmark dataset (NOMO3D) to prove efficiency of the developed algorithm. It is observed from the results, the developed method could provide success rates from 28% to 93% for male and from 24 to 82% for female subject depending on the measurement. Félix et. al. have developed depth texture synthesis of large scenes based on geometry of 3D meshes of large scenes with such repeating elements [10]. The authors, have utilized the RGB

and SfM depth information as a guide. Consequently, the developed approach extends the high-resolution mesh by exploiting powerful, image-based texture synthesis. The authors pointed that the proposed approach benefits from reduced manual labor as opposed to full RGBD reconstruction, and can be done much more cheaply than with LiDAR-based solution. Image processing based separation and counting systems has become popular in industry during the recent years. Ponce et . al. have developed an image processing based olive fruits detection system [11]. The developed system can not only detect olives but also compute size and mass. The developed model has relative errors below 0.80% and 1.05% for the estimation of the major and minor axis length for all varieties, respectively. Doğan et. al. have developed a Kinect based ground plane detection system [12]. Arvapally et. al. have developed a FPGA based industrial bottle counter system [13]. Bottle was detected by an infrared sensor which was mounted on the conveyor belt. The output of the sensor was processed by an ARTIX-7 NEXYS-4 FPGA board. The authors pointed that the developed system is suitable for applications for which high precision and speed is not required. Wenju et. al. have developed an image processing based cap defect detection system [14]. The developed system utilizes the circular region projection histogram (CRPH) as the matching features. Hui-Min et. al. have developed an image processing based bottle defect detection system [15]. The developed systems consists of eight video cameras installed beside the production line to capture the images of the mouth, lip and neck of each bottle. Wang et. al. have developed a bottle detection system for UAV’s [16]. The authors have compared the developed algorithms performance several stateof-the-art object detection algorithms on the UAV-Bottle Dataset (UAV-BD), such as Faster R-CNN, SSD, YOLOv2 and RRPN. The professional bottle counting systems placed on the market are generally based on ultrasonic or infrared sensors mounted on the production line [17-18]. In these systems, the presence of a bottle could be detected according to the sensor output. Therefore, each bottle should independently pass by the sensor for accurate counting. On the other hand, bottles used in beer industry and also in several other industries are typically packed in boxes. Therefore, it is clear that sensor based detection systems are not applicable to beer bottle counting. Alternatively, image processing based bottle counting systems can be employed to overcome this problem [19]. . However, image processing systems based on visible light fail due to changes in illumination, shadows, reflections, bottle impureness, bottle level (fill or empty) and the presence of the bottle cap. In this study, an image processing based bottle counting system is proposed using images acquired from a depth camera. The depth frames are captured by a Kinect v2 camera with 512*424 resolution. On the contrary to the high cost industrial cameras, the Kinect sensor is cheap and easy to implement for the

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

proof of concept applications. The developed system is not only a cost-effective alternative to existing systems, but also provides accurate detection of the bottle. Unlike stereo camera-based systems, measurement accuracy is not affected by changing light conditions unless placed directly under sunlight or infrared light source. The developed system can easily integrate to the re-cycling line with the plug-and-play principle. On the other hand, the developed system does not require any additional camera calibration or light control system despite changing operating conditions. In fact, the company that asked the authors to develop a real-time bottle counting system for its re-cycling facility, have already unsuccessfully tested an image processing based system that employed a high resolution Cognex camera. Although the system was shown to detect bottles with high accuracy in laboratory environments, the detection accuracy was limited to 60% under true facility running conditions. In addition to its low cost, the depth measurements of the Kinect are not affected from the disturbances that mentioned before. There are three different type of bottles in the re-cycling facility the system is developed for. More than four hundred thousand bottles are recycled in the facility per day. It is observed from the results that the developed algorithm could count the bottles with 99% accuracy for each bottle type which is satisfactory for real time operations. The schematic of the proposed system is given in Fig. 1.

Theoretical Background

In this study image processing methods are applied in order to detect and count the bottles. The theoretical background of the utilized methods (camera calibration, min bounding rectangle) are given in this section.

Camera Calibration

Camera calibration, which is a common problem addressed in the literature, is the initial step of the proposed algorithm [20-21]. Calibration is essential to obtain the intrinsic and the extrinsic matrices of the camera. The intrinsic matrix is based on the internal parameters of the camera such as focal length(f), principle point (cx, cy), pixel width(px) and pixel height. Using these parameters, the intrinsic matrix M is defined as follows  − f / sx  M= 0  0 

These parameters are specific to each manufactered Kinect v2 sensor and is provided by the supplier. In this study, manufacturer provided internal parameters are obtained from the Kinect SDK.

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

The transformation between pixel space to real world coordinates can be calculated as follows xk zk i  yk −1   = M  j zk 1   1

where θ and Ø are the rotation angles for x and y axes respectively. Transformation between a point in Kinect coordinate system (pk=[xk yk zk]) and corresponding point in the new coordinate system (pm=[xm ym zm]) can be calculated as follows. pm = Rpk

The equation 4 can be written as linear equations, xm = r11 x k + r12 y k + r13 z k ym = r11 x k + r12 y k + r13 z k

z m = r31 x k + r32 y k + r33 z k = dm where rij is the i-th row and j-th element of the R matrix. The plate depth value dm should be constant for each point on the re-cycling line. Based on this information the third row of the equation 5 can be re-written as r3T pk = dm

For N sample points randomly chosen on the measuring plate, N equations can be written as follows,  pkT1  T  pk 2  T  pk 3 R=        T  pkN

where i and j are row and column indices, respectively. The extrinsic matrix depends on the camera placement. The depth value of a point on the re-cycling line as measured by Kinect varies depending on the location due to the mounting angle between the Kinect and the re-cycling line. Therefore, a coordinate transformation is required between Kinect’s coordinate system and recycling line coordinate system. The Kinect coordinate system should be rotated over x and y axes in order to make this transformation. Rotation in z axes is neglected and not included in the rotation matrix since rotation in z axes does not affect the depth measurements. The combined rotation matrix R around the x and y axes are calculated as follows, cos ( 0 )

where dm is the plate depth of the system. All the points on the re-cycling line should satisfy the equation given above.

Equation 18 can be re-written as Ax = 0 format;  pkT1  T  pk 2  T  pk 3 A=         T  pkN

−1  −1  −1  r3T   d an x =      dm     −1

The values of the A matrix (θ and Ø angles) can be calculated using the singular value decomposition method.

The Minimum Bounding Rectangle Algorithm

The center point of an object (xm, ym) which contains N points can be calculated as follows, xm =

The diagonals of the minimum bounding rectangle must intersect at the object’s center point. The equation of the line, which corresponds the center point is given below xtan (θ ) − y + ym − xmtan (θ ) = 0

where θ is the angle between horizontal and vertical axes at center point. The distance between any edge point and line can be calculated as follows, p (i ) = ( xn − xm ) sin (θ ) − ( yn − ym ) cos (θ )

The sum of the orthogonal distances of all edge points (P) of the object can be calculated as follows, P = ∑ i =1 ( xn − xm ) sin (θ ) − ( yn − ym ) cos (θ ) N

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

The P value must be minimized in order to calculate θ value. Therefore, the first order derivative of the p function according to the θ must be equalized to zero as follows, 2 ∑ i =1 ( x n − x m ) ( y n − y m ) N

Image Processing Algorithm

In this study, depth frames acquired from the Kinect sensor are processed by an image processing algorithm developed in Matlab environment. As an initial step, the calibration algorithm proposed in Section 2.1 is executed to compute the plate depth (distance between camera and the conveyor belt) and the parameters of the camera extrinsic matrix. Using the camera intrinsic matrix obtained from the SDK and the extrinsic matrix computed by the custom calibration routine, the depth images are transformed from camera coordinate system to world frame. The pixels, which are elevated (decreased depth value) by a certain threshold (based on bottle type) and projects on the conveyer belt, are

considered as foreground pixels. All the remaining pixels are assumed as background pixels. Algorithm checks the number of the foreground pixels on the virtual detection line in order to detect the objects. If the number of foreground pixels on the virtual box detection line is higher than a pre-determined threshold, it is assumed that there is an object on the conveyor belt. If the algorithm detects an object, morphological operations are applied to the binary image in order to fill the gaps. The size of largest object is determined by the connected component labeling algorithm. The algorithm starts the measurement process only if the size of the largest object is consistent with the boxes in the database. The sample image and detected box are given in the Fig. 2 The box orientation and box edges are determined by the minimum bounding rectangle method. The box is rotated clockwise according to the orientation angle in order to obtain zero-degree orientation. The original and rotated image and minimum bounding rectangle of box are given in Fig. 3. Box points (including edges) are projected on the conveyor belt surface. Then, the box region is gridized based on the template to determine bottle locations. The templates are given in the Fig. 4.

Figure 2. The Sample image and detected box. Table 1. Characterization of Reactive Orange 16(Lee,Choi ve ark.;2006)

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

Figure 3. The Original and rotated box and minimum bounding rectangle of the box.

Figure 4. The Box grid templates. If the number of pixels that fell in a computed bottle location is higher than a pre-determined threshold, the algorithm assumes that a bottle is present at that location. The cycle time for the developed image processing scheme is 45 ms which is satisfactory for real time applications. The flowchart of the proposed algorithm is given in Fig. 5.

Results And Discussions

In this section, the results of the developed image processing algorithm are provided. Before testing the system real time on the facility re-cycling line, a custom made platform is developed to acquire depth images and build a sample dataset for offline analysis. The image processing algorithm was developed by using this dataset. Fig. 6 and 7 depict the results for a type 1 box that contain 20 bottles. The first box does not have missing bottles whereas the second has 9 missing bottles. The first column shows the raw depth image. The second column illustrates the boxes detected along with the computed bottle locations. Finally,

the third column depicts the filtered box image in which missing bottles can easily be detected. The proposed algorithm could detect not only missing bottles but also missing bottle locations with %100 accuracy. Fig. 8 and 9 depict the results for a type 2 box that contain 30 bottles. The two boxes have 8 and 15 missing bottles, respectively. In Fig. 8, 50% of all bottles are missing but still the algorithm is successful in detecting these missing bottles. Although type 3 boxes contain 20 bottles similar to type 1 boxes, the bottles are thinner. Therefore, the parameters of the developed algorithm are optimized to detect a specific bottle type. Fig. 10 and 11 depict the results for a type 3 box. The sample two boxes have 8 and 15 missing bottles, respectively. As in previous cases, the algorithm achieved a 100% missing bottle detection rate. After developing the algorithm based on the sample dataset, the algorithm is tested over the real time data captured from the re-cycling facility line. In Fig 12, a sample snapshot is given, which is taken from the re-cycling facility

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

Figure 6. Results for sample image for bottle type 1 with 20 bottles.

Figure 7. Results for sample image for bottle type 1 with 11 bottles.

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

Figure 8. Results for sample image for bottle type 2 with 22 bottles.

Figure 9. Results for sample image for bottle type 2 with 15 bottles.

Figure 10. Results for sample image for box type 3 with 17 bottles.

Figure 11. Results for sample image for box type 3 with 14 bottles.

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

processing type 1 boxes. Detection results is provided on the right side of the Fig. 12. The green circles depict the locations where bottles are detected whereas red circles depict the missing bottle locations. The algorithm is robust against disturbances such as varying light conditions, dirty bottles and bottle impureness. On the other hand, if the box contains extra bottles layed horizontally, the algorithm assumes that there is bottle present under the related locations as shown in Fig. 13. During the experiments it is observed that the algorithm could only fail in the absence of enough distance between two consecutive boxes. As it given in Fig. 14., the developed bottle detection algorithm fails due to incorrect box detection. The gathered real time

Figure 12 .Algorithm result for bottle type 1( real time data).

Figure 13 Algorithm result for extra bottles (real time data).

depth data can be downloaded from link given below. https://drive.google.com/drive/folders/14OU0tnrAsQ8X0K57QWebsUShOQJ59ws?usp=sharing. The depth data is the .mat format and easy for implement for the researchers who desires to focus in this topic. The performance of the developed algorithm on the gathered data could be downloaded from the link given below. As it can be seen from the video, the developed algorithm failed to count only one missing bottle. Therefore, it is proven that the algorithm has over than %99 missing bottle counting accuracy. The performance tests of the algorithm is not evaluated under the light controlled environment. Therefore, the disturbance of the light variations exist for all gathered depth images. The effect of the

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

Figure 14. Algorithm result for two consecutive boxes (real time data). Table 1. Effects of the each disturbance on the algoritm (Total 12039 bottles) Disturbance Type

remaining disturbances bottle impureness, presence of the bottle cap and foreign objects in box is analyzed with the Table 1.

Conclusions And Future WORK

In this study, a Kinect v2 based missing bottle detection system for a re-cycling facility is proposed. Kinect v2 camera is utilized since its depth measurements are robust even in the presenece of disturbances (light variation, bottle impureness, etc.). Image processing algorithm is developed in Matlab environment by using an offline sample dataset constructed from frames acquired from a custom set-up. The parameters of the algorithm are optimized during the offline analysis of the sample dataset. The developed algorithm is tested over the real time depth images gathered from the facility re-cycling line. It is observed that the developed algorithm has over 98% missing bottle detection accuracy. Consequently, it is proven that missing bottle accuracy of the developed algorithm is sufficient enough for the facility to plan the new production. In addition to high detection accuracy, computational time is another important parameter for the developed system. Processing time for a single depth frame is 45 ms which is fast enough for real time applications. Although the Kinect has lower price than traditional depth cameras, it is not suitable for the industrial applications due to its operational performance. It is observed that Kinect V. 2 camera could not

work properly over than ten hours. Therefore, at least two Kinect cameras should be utilized as alternate of each other for this system. Similarly, the humidity of the facility could affect the Kinect depth measurements. Therefore, an industrial depth camera could be used instead of the Kinect camera

Acknowledgements

The authors thanks the ROMEDA LTD and Tuborg A.Ş. for the support of this study. The performance video of the system, which contains the obtained results while developed code is running, could be downloaded from the link given below. https://drive.google.com/open?id=1t9MvW8__ GjlVlQ51HcJ3pzc7royObF1b.

Data Availability Statement

The authors confirm that the data that supports the findings of this study are available within the article. Raw data that support the finding of this study are available from the corresponding author, upon reasonable request.

Sigma J Eng Nat Sci, Vol. 40, No. 3, pp. 464–474, September, 2022

Conflict Of Interest

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics

There are no ethical issues with the publication of this manuscript.

Share and Cite