Educational & Group project
November 2023 – Current
Overview
Originally the project started as the second block semester project of the 2nd year of my Bachelor’s program. In collaboration with the Netherlands Plant Eco-Phenotyping (NPEC) Center, our mentor Dr. Alican Noyan Mehmet created 14 tasks that when combined would result in a pipeline that would use a U-Net model to segment roots, measure them then use PID and RL based approaches to get a simulated robot to inoculate the primary root tips.
At the end of the 8 week educational project, our lecturer organized a Kaggle Competition where me, and 4 other colleagues, ranked top 5 in the competition were selected to continue working on our solution and write a research paper to showcase our results and aid NPEC in publishing their paper on their HADES robot.
My main responsibilities in the second part of the project were developing a robotics pipeline to automate the inoculation process based on the output of the model and assuring the integration of multiple codebases into one singular solution.
Dataset
For the initial phase of the project we had 3 datasets. The main dataset consisted of 126 images that were labeled by students under supervision from plant biologists from Utrecht University. Each student labeled 2 to 3 images depending on the complexity. Each image contains 5 images of plants in a Petri dish at various stages of growth.
Each image got 4 masks created: root, shoot, seed and occluded_root; the later proved to not be needed as the model learned to fill in small gaps by itself and we would end up developing our own solution for fixing occlusions.
In the second phase our project, we received more data labelled by plant biologists from Utrecht University and to our training pipeline added open source images from a computer vision competition with the same scope as us.
When asking for more data we took into account the disproportionate amount of images we had for the 4 to 8 weeks of growth in comparison to ungerminated and small plants which had an impact on our initial results.
Approach
For ease of understanding the project can be separated into three separate parts, in this order: Segmentation -> Processing -> Robotics
Segmentation
After tackling the particularities of the U-Net model input and output standards and working with large datasets on our local machines during the initial phase of the project, the following variations of the model used combinations of hyper parameters alongside data augmentations to “nudge” the model in the right direction.
Processing
Besides the typical post-processing of model outputs, we also measure the plants and individually segment the roots for each plant individually, which increases complexity in the many cases where roots grow to the point of overlapping on top of each other. Through graph network analysis we obtain not only the length in pixels but also the landmarks for inoculation such as primary root tips, lateral root tips and junction points which allows us to proceed to the next step in the pipeline.
Robotics⭐️
Besides my role in assuring the smooth integration of our collective codebases and the modularity of the code for our deployment and future expansion, my main responsibility was developing the robotics solution for our pipeline. NPEC is in possession of two Opentrons OT2 lab robots which they intend to use in the future inside HADES to automate tasks such as inoculation.
The initial development of the solution was done remotely by connecting to the Jupyter server hosted on the robot and a webcam, I was able to develop an initial prototype.
During the mandatory part of the project, we had a simplified simulated PyBullet Environment of the robot, which we used to learn about developing both PID and Reinforcement Learning solutions. For this specific use case where the only changing variables are the coordinates of inoculation, PID proved both easier to develop, more accurate and faster.
Currently we are using the package distribution of Opentrons in order to create a custom protocol, we interface with the Jupyter server of the robot via API calls in order to pass coordinates to the robot and information about the reservoirs to the pipeline.
Research Paper
We are currently in the process of writing the first draft of the manuscript for the research paper. The scope and the complexity of the problem is always changing and so we find ourselves wanting to include increasingly more information in the paper while still keeping the technical details to a minimum due to our aim to publish in a more biology focused journal.
Results
Pipeline results
Robotics⭐️
matachevlad.com 