Epidemiological Arbovirus Classifier

Initial state of the SINAN data

One important point first: there is a term called "GIGO", short for "garbage in, garbage out", which captures how much a consistent treatment of the SINAN data matters. Before treatment, the data had several problems that would get in the way of both data analysis and a sound machine learning pipeline. Among them were many columns with no use for prediction.

Some of these columns could "leak" the classification, while others simply would not help because of what the column represents. There were also empty fields or fields marked as ignored, each year showed a different pattern in some columns, and after 2021 there is no data with the "Discarded for arbovirus X" classification, which is why data was only extracted, and used for testing, up to that year.

Selecting the years

• The years 2014 to 2016 had missing or incomplete symptoms.
• Using them could teach the model differences in how the forms were filled out, rather than differences between confirmed and discarded cases.
• For that reason, training used the most consistent data, from 2017 to 2019.
• The year 2020 was set aside to tune the models' decisions.
• The year 2021 was held out for the final evaluation.

Case classification

• The original classification codes were simplified into two groups: confirmed and discarded.
• Cases without a conclusion or with an inconclusive classification were not used to train the models.
• That way, every record used had a clear answer for what the model was supposed to learn.

Symptoms

• Symptoms were represented as present, absent or not reported.
• “Did not have a fever” was kept distinct from “we do not know whether they had a fever”.
• Besides the individual symptoms, the number of symptoms present was calculated.
• The number of symptoms actually filled in was also recorded.
• Combinations were created to represent symptoms that appear together.
• Rash and retro-orbital pain received a specific summary because they showed relevant differences between confirmed and discarded cases.

Time and seasonality

• The number of days between symptom onset and notification was calculated.
• Month and epidemiological week were included because dengue circulation changes throughout the year.
• This information was turned into cyclical variables.
• This was needed because the calendar is a cycle: December is close to January, even though 12 looks far from 1 numerically.
• The same logic was applied to epidemiological weeks.

Demographic and location information

Demographic characteristics are a very important part of the treatment. Education levels were grouped into similar levels and arranged on an increasing scale, so the variable represents a progression from missing or unreported schooling up to higher levels such as college, master's and doctorate.

• Columns such as sex, race and pregnancy were transformed with one-hot encoding, which avoids implying an unrealistic progression
• Since there were many occupations, following the CBO, those with few members were grouped into "Other"

Location data was kept at three levels: state, municipality and health region. This is useful because different places have characteristics that can make it easier for these arboviruses to spread and infect people.

Epidemiological context

One of the most interesting transformations in the project was representing what was happening in the municipality before each notification. This helps capture the local "recent scenario" of arbovirus circulation, and two main pieces of information were created for it:

• Volume of notifications in the municipality over the previous four weeks
• Share of confirmed cases over the same period

The idea is that a person's symptoms can mean different things depending on recent dengue circulation in that area. Only previous weeks were considered, without using the patient's own result. This point deserves its own section because it connects data treatment to the project's epidemiological logic, and it was very useful for increasing the models' overall predictive power.

From treated data to the training set

During the exploratory analysis, we saw that no single group of information was enough to represent a case's classification on its own. Symptoms matter, but their meaning also changes when they are seen alongside the time of year, the place of notification and the recent context of the municipality. So I brought these perspectives together into a single set of 107 features, which became the common input for the Machine Learning models.

Training used 2,874,235 records from 2017 to 2019. For each row, the model received the information available about the notification and tried to tell confirmed cases from discarded ones. Test results, clinical progression, case closure, hospitalization and any other later information about the patient were deliberately left out of the features: if they were used, the model could learn the very answer it was supposed to predict, creating an ability that would not exist in a new screening.

I also kept the same feature schema throughout the whole pipeline. This means the order, names and treatment of the variables used in training are the same ones expected at prediction time, which prevents the models from receiving information organized one way during training and another way in use.

Why use three different models?

With the training set ready, I could have picked a single algorithm and pushed it as far as possible, but different models can pick up different patterns in the same information. So I chose an MLP neural network and two tree-based gradient boosting models, XGBoost and LightGBM. All three receive the same 107 features, but they learn the relationships in the data in different ways.

This difference is useful because the three models do not see the data the same way, so they do not have to make exactly the same mistakes. The MLP can find non-linear interactions among many variables at once, while XGBoost and LightGBM build sequences of trees that split the records according to the patterns they find. Each architecture can therefore contribute a different way of understanding the same problem.

How does the MLP learn from tabular data?

Neural networks are often associated with images or text, but they can also be used with tabular data. In this project, I built an MLP that handles categorical and numerical information separately. Education level, occupation, state, municipality and health region were turned into embeddings, smaller representations that let the network learn similarities between categories during training, instead of treating each code as just an ordinary number.

The numerical features were normalized and then joined with the embeddings before entering the network's four hidden layers, with 1024, 512, 256 and 128 neurons. Between these layers I used LeakyReLU, Batch Normalization and Dropout, which help the network learn complex relationships, keep training stable and reduce the risk of memorizing the training records. Below is the main part of the code used to build this structure:

To train the MLP, I used the AdamW optimizer and a loss function that takes the difference between the classes into account. Learning was monitored on the 2020 validation set and stopped when the network stopped improving. As a result, the saved model was not simply the one from the last epoch, but the one that performed best on validation.

XGBoost and LightGBM: two tree-based perspectives

XGBoost and LightGBM belong to the gradient boosting family. Unlike a single decision tree, these models build trees in sequence, with each new step trying to correct part of the previous errors. This is especially useful with tabular data, where symptoms, dates, occupations and locations can form many combinations that are hard to capture with isolated rules.

Even though both use the same principle, they do not build their trees in exactly the same way: XGBoost offers a more controlled regularization and growth process, while LightGBM was designed to train efficiently on large datasets. Since the project has millions of records and over a hundred features, comparing the two implementations showed how these differences played out in classifying cases. Below is the part of the code that creates each model:

To choose the hyperparameters, I used Optuna on stratified samples of the training and validation data. It tested different combinations of number and depth of trees, learning rate, share of rows and features used, and regularization levels. After finding the best configuration, each model was retrained on the full set, always using 2020 to monitor learning and apply early stopping when new trees stopped bringing improvements.

The role of temporal validation

Throughout this process, 2020 played a different role from the training years. Data from 2017 to 2019 was used to teach the models the patterns, while 2020 was used to compare configurations, stop training at the right moment and define how the predictions would be combined. These decisions were thus made by looking at a period after training, but without using the set reserved for the final evaluation.

This separation mattered because a random split would mix records from different years between training and validation. Since dengue circulation, the way notifications are filled out and the epidemiological context itself change over time, a random split could show a better result than what would actually happen in a future period. That is why I respected the order of the years, making the pipeline learn from the past and then classify new notifications.

Only after finalizing the models, their hyperparameters, the ensemble weights and the classification rule was the 2021 set loaded for evaluation. Its 940,304 records took no part in any earlier choice, so the final result represents an evaluation outside the period used to build the system.

Combining the predictions into an ensemble

After training, each model produced its own score for a case. Instead of a simple vote, I combined these three values with a weighted average. The weights were set from each model's recall on the 2020 validation set and then normalized, so the models that found more confirmed cases in that period had a slightly larger share of the combined result. Below is the part of the code that performs this combination:

The idea of the ensemble was to take advantage of the differences between the models. When MLP, XGBoost and LightGBM find a similar pattern, the combined score has the support of all three; when they disagree, the weights help balance the predictions according to how each model behaved on validation.

In the end, this process delivers a single classification score, which still has to be compared with a threshold to decide between confirmed and discarded. This value is not a calibrated clinical probability and does not make a diagnosis. The choice of operating point and its impact on hits and errors are analyzed in the next part, along with the model's metrics.

Opening the final 2021 test

The evaluation covered 940,304 notifications: 484,586 discarded cases and 455,718 confirmed cases. The two groups are close in size, and since the records come from a year after the training period, the test is closer to a real situation, in which the system learns from past data and then receives new notifications in the future.

At this point, no hyperparameter was changed, the models were not retrained and the 2021 result was not used to choose the threshold. So the metrics do not show the best fit that could be found by looking at the test itself, but the behavior of the pipeline that was already defined before these records were opened.

Accuracy shows that the ensemble got just over seven out of ten classifications right, but this value should not be read alone: the goal of the chosen threshold was not to spread errors evenly, but to reduce the number of confirmed cases that would be missed. That is why Precision and Recall need to be read together to understand which type of error the system was designed to accept.

With a Precision of 66.1%, some of the notifications classified as dengue are still discarded cases, while the Recall of 87.3% shows that most of the actually confirmed cases were found. This balance makes sense for a screening support tool, since letting a true case slip through can be more worrying than sending an extra case for evaluation, even though it also increases the number of false positives.

Why does the 0.30 threshold change how we read the model?

The ensemble does not directly output the words confirmed or discarded: it produces a continuous score, and the threshold turns that value into a classification. When the cutoff is lowered, more records are flagged as positive, Recall tends to increase and fewer confirmed cases fall below the cutoff. However, Precision and specificity may drop, because more discarded cases are also sent to the positive group.

The 0.30 threshold was set using only the 2020 validation set and stayed frozen before the final test, while the individual models kept their own thresholds chosen on the same period. So the 2021 evaluation was not used afterwards to look for the cutoff that would produce the most favorable metric.

There is a region where different thresholds produce similar F1 values, so the single highest value of this metric does not have to be the only decision rule. For the ensemble, the 0.30 point keeps a result close to the highest values and increases sensitivity, a choice consistent with the goal of finding more confirmed cases before the final evaluation.

Reading hits and errors in the confusion matrix

Metrics summarize the model's behavior, but the confusion matrix shows where each percentage comes from, separating correct predictions on discarded and confirmed cases from the two possible types of error. Instead of looking only at Accuracy, we can see how many records the chosen threshold pushed to each side.

The key reading lies in the relationship between the row of confirmed cases and the row of discarded cases: the bottom-right quadrant shows the confirmed cases found, while the top-right one gathers the discarded cases that were also sent to the positive class. This shows why increasing Recall can also increase false positives.

This trade-off needs to be clear to whoever uses the result: the system accepts sending more records to the positive group in order to reduce the number of confirmed cases below the threshold. This behavior can help with prioritization, but it should never be interpreted as an automatic diagnosis.

Was the ensemble better on every metric?

Comparing the models shows that there is more than one way to decide which one was best. XGBoost and LightGBM reached slightly higher Accuracy and Precision at their own operating points, while the ensemble, using the 0.30 threshold, gave up some of that precision to find a much larger share of the confirmed cases.

The chart shows that the models reached similar results on several metrics, while the differences grow on those that depend on the cutoff. The main role of the ensemble is therefore not to beat the other models on every result, but to combine their predictions to deliver the behavior chosen for screening.

The ability to separate the two groups

ROC-AUC looks at the model across many thresholds, comparing the true positive rate with the false positive rate. It does not just show the decision made at 0.30, but the overall ability to rank confirmed cases above discarded ones at different cutoffs.

The nearly overlapping curves show that the models have a similar ability to separate the groups, and that the most visible difference appears when the scores are turned into classes by a threshold, while the Precision-Recall curve shows the trade-off between finding more confirmed cases and keeping more confidence in the positive flags.

It is also important not to confuse ROC-AUC with Accuracy or with clinical probability. This metric summarizes the ranking produced by the score across thresholds; it does not mean that the same percentage of patients will be classified correctly, nor that the value produced represents a calibrated individual risk.

From notification to model scores

The Screening page brings together the information available at the time of notification, such as age, sex, pregnancy, race, education level, occupation, location, dates and symptoms. You can also choose which arbovirus will be evaluated. The interface organizes the fields sent to the backend and does not depend on later information, such as test results, clinical progression or case closure.

The interface was built with React and TypeScript, while the FastAPI API receives the fields, applies the same preparation used during training and creates the features the models expect. Then MLP, XGBoost and LightGBM produce their individual scores, which the ensemble combines before the response goes back to the browser.

Presenting the result without turning it into a diagnosis

In the response, the three scores appear separately along with the weights used in the ensemble, and right below, the application shows the combined classification and the threshold applied. This makes it clear that the final result does not come from a single algorithm, and shows when the models agree or disagree.

Besides manual screening, the Home page lets you draw a random anonymized case from the test set and compare the prediction with the classification recorded in the historical database. This simulation is meant to demonstrate the full pipeline, not to clinically validate anyone, so the application presents the result as a classification score and keeps the notice that the tool does not replace evaluation by a health professional.

Showing the data journey too

So that the application would not be limited to the form, I created a Pipeline page that explains the path from SINAN data through preparation, training, evaluation, saved models and the API. I also created an Epidemiological Overview page that brings together the main findings of the exploratory analysis, so visitors can understand both the result shown and the whole process behind it.