#
Epochs & TensorBoard
Last update: Feb 10, 2024
#
#
Epochs
"Epoch" is a unit of measuring the training cycles of an AI model.
In other words, the amount of times the model went over its dataset and learned from it.
#
#
How many epochs should I use for my dataset?
There isn't a way to know the right amount previous to training. It depends on the size, length & quality of the dataset.
If you aim towards a quality model, it's not convenient to input a semi-arbitrary amount of epochs, as it makes it prone to underfitting/overtraining. (explained later)
So it's best to use TensorBoard. WIth it you can determine exactly for how long you should train. (explained later)
#
#
Do more epochs equal a better model?
- It doesn't, since using a disproportionate amount will overtrain the model, which will affect the quality of it.
#
#
Overtraining
#
In the field of AI, is when an AI model learns its dataset too well, to the point where it centers too much around it & starts replicating undesired data.
The model performs very well with data of the dataset, but poorly with new data, as it has lost its ability to replicate anything that deviates from it.
It happens when the model is trained for too long/is too complex. So to avoid this, RVC users use a tool called TensorBoard.
#
#
TensorBoard
#
TensorBoard is a tool that allows you to visualize & measure the training of an AI model, through graphs & metrics.
It's specially useful for determining when to stop training a voice model, since with it you can detect when the overtraining point begins.
Because of this, TB is the most convenient tool for RVC users for perfecting a voice model.
#
#
Installing & Opening
For RVC Disconnected users, ignore this, it opens up by itself when you start training.
#
Download this file & move it inside RVC's folder. Ensure the file path doesn't contain spaces/special characters.
#
- Now execute it. It will open a console window & create some folders inside RVC.
- If you get the
Windows protected your PCissue, click More info & Run anyway.
- If you get the
- Once it's done, your default browser should open with TensorBoard app.
- If it doesn't, copy the address of the console at the bottom, and paste it in your browser.
Said address will say "https://localhost" followed by some numbers.
- If it doesn't, copy the address of the console at the bottom, and paste it in your browser.
#
#
Usage Guide
#
#
SETTING UP
Open TB & begin training in RVC.
If you get the
No dashboards are activeissue, selectSCALARSin the top right corner dropdown.
First ensure auto-refresh is on, so the graphs update constantly.
Click the gear () in the top left corner & turn on
Reload data.
You can always manually refresh with the refresh symbol (🔄) in the top right.
Go to the
SCALARStab.
#
GRAPH
# In the left panel:Activate
Ignore outliers in chart scaling.Set Smoothing to
0.987.Select your model in the
Runssection below. The models you tick will show in the graphs. (untick/evalif you want)
- In the search bar, type "g/total". This will be the graph you'll monitor.
- Each graph has three buttons in the corner:
- Left one is for going fullscreen.
- Middle one to disable Y axis, for a fuller view.
- And the right one is to center the view.
- To zoom in & out the graphs, press the ALT key + mouse wheel. Remember to center the view after moving around, and after the graph updates.
#
MONITORING
Now let the training go for some time.
You'll detect OT (overtraining) when the graph hits the lowest point, then stay flat/rising indefinitely.
Example of OT for hours:
There will be various low points, one after the other, so don't get too anxious if it's OT or not. You can always use a previous checkpoint either way.
If it reaches a low point, let it run for longer until it's very clear it's OT.
When you detect OT, zoom in & place your mouse over the lowest point. Note down the
Stepnumber.
Use the model that was closest to before the OT point.
For more information, check the guide for your RVC in this website.
~ Work in progress. ~