This document describes how to install RecStudio, train and evaluate models built into RecStudio.
Clone RecStudio
To use RecStudio, you are required to clone RecStudio source code from our repo.
git clone https://github.com/ustcml/RecStudio.git
cd RecStudio
Configure the environment
Set up the environment required to run RecStudio.
Conda
Run the following code to set up the conda environment.
conda env create -f environment.yml
Run RecStudio
The following is an example of running a built-in model (BPR) of Rec Studio. If you want to build the model modularly, please follow the link. Click the link if you want to run a custom algorithm.
Below is an example of running a dataset (ml-100k) built into RecStudio. If you want to use a non-built-in dataset, please refer to the link.
Run in commond line
We provide the run.py script for users to quickly use the built-in models.
python run.py -m=BPR -d=ml-100k
The above use case trains and evaluates the BPR model on the ml-100k dataset. You should get an output like this:
[2022-11-05 14:22:19] INFO Global seed set to 2022
[2022-11-05 14:22:19] INFO dataset is read from recstudio/dataset_demo/ml-100k.
[2022-11-05 14:22:20] INFO
Dataset Info:
=============================================================================
item information:
field item_id
type token
## 1683
=============================================================================
user information:
field user_id age gender occupation zip_code
type token token token token token
## 944 62 3 22 795
=============================================================================
interaction information:
field user_id item_id rating timestamp
type token token float float
## 944 1683 - -
=============================================================================
Total Interactions: 100000
Sparsity: 0.936953
=============================================================================
[2022-11-05 14:22:20] INFO
Model Config:
accelerator=gpu
ann=None
batch_size=2048
cutoff=[10, 20, 5]
dataset_neg_count=None
dataset_sampler=None
early_stop_mode=max
early_stop_patience=10
embed_dim=64
epochs=100
eval_batch_size=20
excluding_hist=False
fmeval=False
gpu=[0]
init_method=xavier_normal
init_range=None
item_bias=False
learner=adam
learning_rate=0.001
negative_count=1
num_threads=10
num_workers=0
sampling_method=none
save_path=./saved/
scheduler=None
seed=2022
shuffle=True
split_mode=user_entry
split_ratio=[0.8, 0.1, 0.1]
test_metrics=['recall', 'precision', 'map', 'ndcg', 'mrr', 'hit']
topk=100
val_metrics=['ndcg', 'recall']
weight_decay=0
[2022-11-05 14:22:20] INFO save_dir:./saved/
[2022-11-05 14:22:20] INFO BPR(
(loss_fn): BPRLoss()
(score_func): InnerProductScorer()
(item_encoder): Embedding(1683, 64, padding_idx=0)
(query_encoder): Embedding(944, 64, padding_idx=0)
(sampler): UniformSampler()
)
[2022-11-05 14:22:27] INFO Training: Epoch= 0 [ndcg@10=0.0095 recall@10=0.0082 train_loss_0=0.6933]
[2022-11-05 14:22:27] INFO Train time: 0.88459s. Valid time: 0.44923s
[2022-11-05 14:22:27] INFO ndcg@10 improved. Best value: 0.0095
[2022-11-05 14:22:27] INFO Training: Epoch= 1 [ndcg@10=0.0165 recall@10=0.0117 train_loss_0=0.6914]
[2022-11-05 14:22:27] INFO Train time: 0.10031s. Valid time: 0.31595s
[2022-11-05 14:22:27] INFO ndcg@10 improved. Best value: 0.0165
[2022-11-05 14:22:27] INFO Training: Epoch= 2 [ndcg@10=0.0625 recall@10=0.0442 train_loss_0=0.6866]
[2022-11-05 14:22:27] INFO Train time: 0.06756s. Valid time: 0.22694s
[2022-11-05 14:22:27] INFO ndcg@10 improved. Best value: 0.0625
[2022-11-05 14:22:28] INFO Training: Epoch= 3 [ndcg@10=0.1222 recall@10=0.0947 train_loss_0=0.6677]
[2022-11-05 14:22:28] INFO Train time: 0.06872s. Valid time: 0.27523s
[2022-11-05 14:22:28] INFO ndcg@10 improved. Best value: 0.1222
[2022-11-05 14:22:28] INFO Training: Epoch= 4 [ndcg@10=0.1496 recall@10=0.1212 train_loss_0=0.6144]
[2022-11-05 14:22:28] INFO Train time: 0.06716s. Valid time: 0.08444s
[2022-11-05 14:22:28] INFO ndcg@10 improved. Best value: 0.1496
[2022-11-05 14:22:28] INFO Training: Epoch= 5 [ndcg@10=0.1611 recall@10=0.1336 train_loss_0=0.5304]
[2022-11-05 14:22:28] INFO Train time: 0.06559s. Valid time: 0.41562s
[2022-11-05 14:22:28] INFO ndcg@10 improved. Best value: 0.1611
...
...
...
[2022-11-05 14:22:51] INFO Training: Epoch= 65 [ndcg@10=0.2187 recall@10=0.2065 train_loss_0=0.2029]
[2022-11-05 14:22:51] INFO Train time: 0.14427s. Valid time: 0.48186s
[2022-11-05 14:22:52] INFO Training: Epoch= 66 [ndcg@10=0.2190 recall@10=0.2073 train_loss_0=0.2032]
[2022-11-05 14:22:52] INFO Train time: 0.10041s. Valid time: 0.65180s
[2022-11-05 14:22:52] INFO Training: Epoch= 67 [ndcg@10=0.2193 recall@10=0.2100 train_loss_0=0.2005]
[2022-11-05 14:22:52] INFO Train time: 0.08839s. Valid time: 0.17648s
[2022-11-05 14:22:52] INFO Early stopped. Since the metric ndcg@10 have not been improved for 10 epochs.
[2022-11-05 14:22:52] INFO The best score of ndcg@10 is 0.2261 on epoch 57
[2022-11-05 14:22:52] INFO Best model checkpoint saved in ./saved/BPR-ml-100k-2022-11-05-14-22-19.ckpt.
[2022-11-05 14:22:52] INFO Best model checkpoint saved in ./saved/BPR-ml-100k-2022-11-05-14-22-19.ckpt.
[2022-11-05 14:22:53] INFO Testing: [recall@10=0.2464 precision@10=0.1920 map@10=0.1645 ndcg@10=0.2779 mrr@10=0.4451 hit@10=0.7869 recall@20=0.3574 precision@20=0.1484 map@20=0.1574 ndcg@20=0.2976 mrr@20=0.4512 hit@20=0.8727 recall@5=0.1523 precision@5=0.2318 map@5=0.1865 ndcg@5=0.2707 mrr@5=0.4265 hit@5=0.6458]
You can modify the parameters to choose different models and datasets. E.g:
python run.py -m=FM -d=ml-1m
The models built in RecStudio are listed in the link, and the datasets built in RecStudio are listed in the link.
Run in .py file
You can use the quickstart module to run a built-in model:
from recstudio import quickstart
quickstart.run(model='LR', dataset='ml-100k', gpu=[2])
The above use case trains and evaluates the LR model on the ml-1m dataset. You should get an output like this:
[2022-11-05 15:06:22] INFO Global seed set to 2022
[2022-11-05 15:06:22] INFO dataset is read from recstudio/dataset_demo/ml-100k.
[2022-11-05 15:06:23] INFO
Dataset Info:
=============================================================================
item information:
field item_id
type token
## 1683
=============================================================================
user information:
field user_id age gender occupation zip_code
type token token token token token
## 944 62 3 22 795
=============================================================================
interaction information:
field user_id item_id rating timestamp
type token token float float
## 944 1683 - -
=============================================================================
Total Interactions: 100000
Sparsity: 0.936953
=============================================================================
[2022-11-05 15:06:23] INFO
Model Config:
accelerator=gpu
ann=None
batch_size=2048
cutoff=[10, 20, 5]
dataset_neg_count=None
dataset_sampler=None
early_stop_mode=max
early_stop_patience=10
embed_dim=1
epochs=100
eval_batch_size=128
excluding_hist=False
fmeval=True
gpu=[2]
init_method=xavier_normal
init_range=None
item_bias=False
learner=adam
learning_rate=0.001
negative_count=None
num_threads=10
num_workers=0
sampling_method=none
save_path=./saved/
scheduler=None
seed=2022
split_ratio=[0.8, 0.1, 0.1]
test_metrics=['auc', 'logloss']
topk=100
val_metrics=['auc', 'logloss']
weight_decay=0
[2022-11-05 15:06:23] WARNING No retriever is used, topk metrics is not supported.
[2022-11-05 15:06:23] INFO save_dir:./saved/
[2022-11-05 15:06:23] INFO LR(
(loss_fn): BCEWithLogitsLoss()
(scorer): LinearLayer(
bias=True
(embeddings): ModuleDict(
(gender): Embedding(3, 1, padding_idx=0)
(item_id): Embedding(1683, 1, padding_idx=0)
(age): Embedding(62, 1, padding_idx=0)
(user_id): Embedding(944, 1, padding_idx=0)
(zip_code): Embedding(795, 1, padding_idx=0)
(occupation): Embedding(22, 1, padding_idx=0)
)
)
)
[2022-11-05 15:06:29] INFO Training: Epoch= 0 [auc=0.5151 logloss=0.8023 train_loss_0=0.8284]
[2022-11-05 15:06:29] INFO Train time: 0.29730s. Valid time: 0.17669s
[2022-11-05 15:06:29] INFO auc improved. Best value: 0.5151
[2022-11-05 15:06:29] INFO Training: Epoch= 1 [auc=0.5387 logloss=0.7607 train_loss_0=0.7806]
[2022-11-05 15:06:29] INFO Train time: 0.09253s. Valid time: 0.13230s
[2022-11-05 15:06:29] INFO auc improved. Best value: 0.5387
[2022-11-05 15:06:29] INFO Training: Epoch= 2 [auc=0.5619 logloss=0.7287 train_loss_0=0.7434]
[2022-11-05 15:06:29] INFO Train time: 0.08588s. Valid time: 0.16471s
[2022-11-05 15:06:29] INFO auc improved. Best value: 0.5619
[2022-11-05 15:06:30] INFO Training: Epoch= 3 [auc=0.5839 logloss=0.7048 train_loss_0=0.7148]
[2022-11-05 15:06:30] INFO Train time: 0.08745s. Valid time: 0.17566s
[2022-11-05 15:06:30] INFO auc improved. Best value: 0.5839
[2022-11-05 15:06:30] INFO Training: Epoch= 4 [auc=0.6051 logloss=0.6870 train_loss_0=0.6936]
[2022-11-05 15:06:30] INFO Train time: 0.08191s. Valid time: 0.11678s
[2022-11-05 15:06:30] INFO auc improved. Best value: 0.6051
[2022-11-05 15:06:30] INFO Training: Epoch= 5 [auc=0.6271 logloss=0.6735 train_loss_0=0.6771]
[2022-11-05 15:06:30] INFO Train time: 0.16541s. Valid time: 0.23597s
[2022-11-05 15:06:30] INFO auc improved. Best value: 0.6271
...
...
...
[2022-11-05 15:06:56] INFO Training: Epoch= 96 [auc=0.7592 logloss=0.5702 train_loss_0=0.5418]
[2022-11-05 15:06:56] INFO Train time: 0.08654s. Valid time: 0.11878s
[2022-11-05 15:06:56] INFO auc improved. Best value: 0.7592
[2022-11-05 15:06:56] INFO Training: Epoch= 97 [auc=0.7593 logloss=0.5701 train_loss_0=0.5421]
[2022-11-05 15:06:56] INFO Train time: 0.08281s. Valid time: 0.23276s
[2022-11-05 15:06:56] INFO auc improved. Best value: 0.7593
[2022-11-05 15:06:56] INFO Training: Epoch= 98 [auc=0.7594 logloss=0.5700 train_loss_0=0.5419]
[2022-11-05 15:06:56] INFO Train time: 0.12219s. Valid time: 0.12029s
[2022-11-05 15:06:56] INFO auc improved. Best value: 0.7594
[2022-11-05 15:06:56] INFO Training: Epoch= 99 [auc=0.7592 logloss=0.5700 train_loss_0=0.5415]
[2022-11-05 15:06:56] INFO Train time: 0.08520s. Valid time: 0.11739s
[2022-11-05 15:06:56] INFO Best model checkpoint saved in ./saved/LR-ml-100k-2022-11-05-15-06-22.ckpt.
[2022-11-05 15:06:57] INFO Testing: [auc=0.7599 logloss=0.5677]
Different models and datasets can be set by modifying the corresponding parameters:
from recstudio import quickstart
quickstart.run(model='BPR', dataset='ml-1m', gpu=[4])
The above use case uses the BPR model trained on the ml-1m dataset, and you can get results like this:
[2022-11-05 15:32:21] INFO Global seed set to 2022
[2022-11-05 15:32:21] INFO Load dataset from cache.
[2022-11-05 15:32:24] INFO
Dataset Info:
==================================================
item information:
field MovieID
type token
## 3707
==================================================
user information:
field UserID
type token
## 6041
==================================================
interaction information:
field UserID MovieID Rating Timestamp
type token token float float
## 6041 3707 - -
==================================================
Total Interactions: 1000209
Sparsity: 0.955316
==================================================
[2022-11-05 15:32:24] INFO
Model Config:
accelerator=gpu
ann=None
batch_size=2048
cutoff=[10, 20, 5]
dataset_neg_count=None
dataset_sampler=None
early_stop_mode=max
early_stop_patience=10
embed_dim=64
epochs=100
eval_batch_size=20
excluding_hist=False
fmeval=False
gpu=[2]
init_method=xavier_normal
init_range=None
item_bias=False
learner=adam
learning_rate=0.001
negative_count=1
num_threads=10
num_workers=0
sampling_method=none
save_path=./saved/
scheduler=None
seed=2022
shuffle=True
split_mode=user_entry
split_ratio=[0.8, 0.1, 0.1]
test_metrics=['recall', 'precision', 'map', 'ndcg', 'mrr', 'hit']
topk=100
val_metrics=['ndcg', 'recall']
weight_decay=0
[2022-11-05 15:32:24] INFO save_dir:./saved/
[2022-11-05 15:32:24] INFO BPR(
(loss_fn): BPRLoss()
(score_func): InnerProductScorer()
(item_encoder): Embedding(3707, 64, padding_idx=0)
(query_encoder): Embedding(6041, 64, padding_idx=0)
(sampler): UniformSampler()
)
[2022-11-05 15:32:33] INFO Training: Epoch= 0 [ndcg@10=0.1086 recall@10=0.0689 train_loss_0=0.6164]
[2022-11-05 15:32:33] INFO Train time: 1.76003s. Valid time: 1.16219s
[2022-11-05 15:32:33] INFO ndcg@10 improved. Best value: 0.1086
[2022-11-05 15:32:34] INFO Training: Epoch= 1 [ndcg@10=0.1170 recall@10=0.0720 train_loss_0=0.3957]
[2022-11-05 15:32:34] INFO Train time: 0.97383s. Valid time: 0.65886s
[2022-11-05 15:32:34] INFO ndcg@10 improved. Best value: 0.1170
[2022-11-05 15:32:36] INFO Training: Epoch= 2 [ndcg@10=0.1353 recall@10=0.0833 train_loss_0=0.3562]
[2022-11-05 15:32:36] INFO Train time: 1.05617s. Valid time: 0.85015s
[2022-11-05 15:32:36] INFO ndcg@10 improved. Best value: 0.1353
[2022-11-05 15:32:39] INFO Training: Epoch= 3 [ndcg@10=0.1450 recall@10=0.0911 train_loss_0=0.3314]
[2022-11-05 15:32:39] INFO Train time: 1.15193s. Valid time: 0.96119s
[2022-11-05 15:32:39] INFO ndcg@10 improved. Best value: 0.1450
[2022-11-05 15:32:41] INFO Training: Epoch= 4 [ndcg@10=0.1506 recall@10=0.0955 train_loss_0=0.3144]
[2022-11-05 15:32:41] INFO Train time: 1.11954s. Valid time: 0.88699s
[2022-11-05 15:32:41] INFO ndcg@10 improved. Best value: 0.1506
[2022-11-05 15:32:42] INFO Training: Epoch= 5 [ndcg@10=0.1573 recall@10=0.0999 train_loss_0=0.3023]
[2022-11-05 15:32:42] INFO Train time: 0.92804s. Valid time: 0.83656s
[2022-11-05 15:32:42] INFO ndcg@10 improved. Best value: 0.1573
...
...
...
[2022-11-05 15:33:39] INFO Training: Epoch= 35 [ndcg@10=0.1887 recall@10=0.1410 train_loss_0=0.2030]
[2022-11-05 15:33:39] INFO Train time: 0.94526s. Valid time: 0.98828s
[2022-11-05 15:33:41] INFO Training: Epoch= 36 [ndcg@10=0.1891 recall@10=0.1407 train_loss_0=0.2000]
[2022-11-05 15:33:41] INFO Train time: 0.91316s. Valid time: 0.84269s
[2022-11-05 15:33:42] INFO Training: Epoch= 37 [ndcg@10=0.1881 recall@10=0.1408 train_loss_0=0.2002]
[2022-11-05 15:33:42] INFO Train time: 0.93156s. Valid time: 0.63566s
[2022-11-05 15:33:44] INFO Training: Epoch= 38 [ndcg@10=0.1872 recall@10=0.1411 train_loss_0=0.1980]
[2022-11-05 15:33:44] INFO Train time: 0.91172s. Valid time: 0.71111s
[2022-11-05 15:33:46] INFO Training: Epoch= 39 [ndcg@10=0.1869 recall@10=0.1402 train_loss_0=0.1971]
[2022-11-05 15:33:46] INFO Train time: 1.05575s. Valid time: 0.79984s
[2022-11-05 15:33:47] INFO Training: Epoch= 40 [ndcg@10=0.1882 recall@10=0.1415 train_loss_0=0.1951]
[2022-11-05 15:33:47] INFO Train time: 0.94636s. Valid time: 0.72500s
[2022-11-05 15:33:47] INFO Early stopped. Since the metric ndcg@10 have not been improved for 10 epochs.
[2022-11-05 15:33:47] INFO The best score of ndcg@10 is 0.1922 on epoch 30
[2022-11-05 15:33:47] INFO Best model checkpoint saved in ./saved/BPR-ml-1m-2022-11-05-15-32-21.ckpt.
[2022-11-05 15:33:47] INFO Best model checkpoint saved in ./saved/BPR-ml-1m-2022-11-05-15-32-21.ckpt.
[2022-11-05 15:33:49] INFO Testing: [recall@10=0.1561 precision@10=0.1864 map@10=0.1336 ndcg@10=0.2353 mrr@10=0.4111 hit@10=0.7313 recall@20=0.2438 precision@20=0.1526 map@20=0.1205 ndcg@20=0.2458 mrr@20=0.4186 hit@20=0.8384 recall@5=0.0957 precision@5=0.2198 map@5=0.1626 ndcg@5=0.2413 mrr@5=0.3933 hit@5=0.6008]