Machine learning (ML) has different stages. It starts with the data collection and preparation, then proceeds to model selection and training. Model evaluation proceeds after the training process is completed. After this, if there are issues encountered, parameter tuning is undertaken.
It is important to point out that monitoring should happen at various stages. It is not limited to a single point of the ML project. However, when exactly do you start monitoring your ML model to examine its performance? At what stage is monitoring too early? When is it too late?
Training stage
Generally, machine learning monitoring starts at the training phase. This phase is when you start feeding verified and normalized information to the system to establish patterns and build its decision-making capabilities. It is expected that the ML system does not immediately achieve the goals set for it. It can produce mistakes or deviations from the expected output or outcomes. Over time, it becomes better at making predictions, which means it improves at performing the tasks it is asked to undertake.
The training phase is the most important part of the ML development process. As such, it should be carefully monitored. However, this monitoring is not limited to checking accuracy and precision. It is equally important to ensure that the data fed to the system is in accordance with the set standards. Most importantly, the data should be accurate and complete. It makes no sense to expect the system to develop the expected capabilities if it is trained with inaccurate and lacking data.
Testing stage
After completing the training process, separate testing using new sets of data is necessary to evaluate the system’s performance independently. This testing process is different from the one conducted during the training process. Monitoring is essential at this stage to ensure the system’s performance is not overfitting or underfitting based on the training data.
Overfitting happens when an ML model follows the training data too closely. This outcome is undesirable because it makes the ML system a clone that follows even the slightest noise from the training data. On the other hand, underfitting takes place when there is almost no learning from the training data. As a result, the system fails to create a suitable model, which means that performance will be remotely accurate, let alone precise.
Deployment stage
Achieving satisfactory results during the testing stage does not guarantee that everything will go as planned at deployment. As such, it is crucial to have monitoring during the deployment stage. The tiniest changes can result in major differences in the system’s accuracy, recall, and other metrics.
It is also important to be meticulous with monitoring during deployment to spot errors or anomalies. A high-profile ML development project is particularly prone to cyber-attacks. There might be attempts to impair the system or introduce changes that make it go haywire.
In summary, ML monitoring should start at the training stage and proceed until deployment. Additionally, there should be ongoing efforts to monitor the ML project as it moves to production. All important metrics input data changes should be tracked.