Supplementary Material

An In-Depth Study on Open-Set Camera Model Identification

Pedro Ribeiro Mendes Júnior, Luca Bondi, Paolo Bestagini, Stefano Tubaro, Anderson Rocha

In this webpage, we present supplementary material for the paper “An In-Depth Study on Open-Set Camera Model Identification”. In Table 1, we present results for all experiments we have performed in this work for all combination of features, training protocols, classifiers, and metrics. Table 2 contains the same results of Table 1 however limited to Open and NetOpen training protocols for the purpose of comparing and highlighting the performance of both of them. In Tables 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, and 14, results presented are limited to WSVM, 2PSVM, DBC, OCSVM, NCM, PSVM, SVM, OSNN, SOFTMAX, SSVM, ET, and PISVM classifiers, respectively. In Tables 15, 16, 17, 18, and 19, results presented are limited to rich, cfa, conv, ip2, and ip1 features, respectively.

Table 1. Open-Set Camera Model Identification (OSCMI) results.
Results obtained for every combination of feature, training protocol, classifier, and metric.
Red, orange, and yellow mark the 3 best results per feature (column), respectively.
Blue indicates the best result per evaluation metric (group of 5 columns).
Bold indicates the best feature per metric (group of 5 columns) and per classifier (row).
Oblique indicates the best result per training protocol (group of 12 rows) and per feature (column).
Metrics NA OSFMM OSFMm FMM FMm DA
Features rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1
Training Protocol Classifiers
Closed WSVM 0.3714 0.4481 0.4760 0.4783 0.4805 0.3321 0.3455 0.3727 0.4063 0.3800 0.2540 0.3053 0.3256 0.3275 0.3270 0.3623 0.3907 0.4115 0.4382 0.4211 0.1563 0.1934 0.2011 0.2006 0.2105 0.2096 0.2177 0.2119 0.2099 0.2221
2PSVM 0.4855 0.5881 0.5371 0.4112 0.4589 0.2339 0.3615 0.3792 0.2895 0.2945 0.1802 0.3479 0.3066 0.2139 0.2227 0.2634 0.3854 0.3997 0.3157 0.3209 0.6556 0.7075 0.5968 0.4304 0.5375 0.6788 0.7130 0.6008 0.4365 0.5439
DBC 0.4683 0.3447 0.5545 0.4936 0.4991 0.1431 0.2163 0.4475 0.4139 0.4052 0.2119 0.2156 0.3614 0.3337 0.3359 0.1818 0.2535 0.4767 0.4526 0.4455 0.5556 0.2185 0.3559 0.2271 0.2373 0.6028 0.2799 0.3650 0.2376 0.2482
OCSVM 0.1631 0.3167 0.5705 0.6467 0.5478 0.0104 0.1202 0.3908 0.4811 0.4175 0.0164 0.0567 0.3127 0.4376 0.2920 0.0602 0.1495 0.4140 0.5083 0.4391 0.2373 0.4455 0.7037 0.8163 0.6683 0.3844 0.5580 0.7197 0.8226 0.6801
NCM 0.0789 0.0924 0.4472 0.4733 0.4572 0.1373 0.1659 0.3499 0.3676 0.3675 0.0541 0.0633 0.3066 0.3245 0.3134 0.1301 0.1572 0.3315 0.3483 0.3482 0.0327 0.0382 0.1850 0.1958 0.1891 0.2069 0.2069 0.2069 0.2069 0.2069
PSVM 0.2490 0.3878 0.4736 0.4779 0.4769 0.2755 0.3865 0.4353 0.4002 0.4110 0.1705 0.2623 0.3234 0.3273 0.3261 0.3050 0.4194 0.4714 0.4417 0.4509 0.1034 0.1728 0.2015 0.1993 0.2011 0.2074 0.2235 0.2144 0.2090 0.2120
SVM 0.5029 0.6272 0.7621 0.7012 0.7612 0.0326 0.1747 0.5524 0.5445 0.5980 0.0122 0.4054 0.5731 0.4653 0.5308 0.0804 0.2134 0.5709 0.5707 0.6183 0.7942 0.8375 0.7729 0.5597 0.6741 0.7943 0.8380 0.7750 0.5676 0.6810
OSNN 0.2525 0.2882 0.4753 0.4799 0.4756 0.2157 0.2837 0.3441 0.3968 0.3629 0.1731 0.1976 0.3259 0.3286 0.3257 0.2444 0.2688 0.3260 0.4387 0.4091 0.1046 0.1192 0.1967 0.2007 0.1983 0.2069 0.2069 0.2069 0.2098 0.2090
SOFTMAX 0.5000 0.5000 0.5000 0.4795 0.5000 0.0000 0.0000 0.0000 0.3833 0.0000 0.0000 0.0000 0.0000 0.3288 0.0000 0.0466 0.0466 0.0466 0.3631 0.0466 0.7931 0.7931 0.7931 0.1984 0.7931 0.7931 0.7931 0.7931 0.2069 0.7931
SSVM 0.5000 0.5353 0.7630 0.7764 0.7993 0.0000 0.0979 0.6385 0.6213 0.6275 0.0000 0.1375 0.5789 0.5515 0.5904 0.0466 0.1685 0.6509 0.6399 0.6453 0.7931 0.8052 0.7866 0.7051 0.7557 0.7931 0.8053 0.7907 0.7124 0.7623
ET 0.3205 0.3737 0.4717 0.4805 0.4757 0.2522 0.2814 0.3652 0.3989 0.3885 0.2197 0.2562 0.3234 0.3287 0.3261 0.2389 0.2666 0.3460 0.4405 0.3680 0.1326 0.1546 0.1951 0.2021 0.1968 0.2069 0.2069 0.2069 0.2114 0.2069
PISVM 0.3544 0.4377 0.4706 0.4753 0.4717 0.3255 0.3838 0.4230 0.4183 0.3912 0.2430 0.3001 0.3227 0.3259 0.3234 0.3084 0.3636 0.4007 0.3963 0.3706 0.1466 0.1811 0.1947 0.1967 0.1951 0.2069 0.2069 0.2069 0.2069 0.2069
Open WSVM 0.5000 0.5079 0.4769 0.4692 0.2181 0.0000 0.0748 0.3934 0.4494 0.5236 0.0000 0.0517 0.3264 0.3215 0.1389 0.0466 0.1295 0.4288 0.4709 0.5167 0.7931 0.7888 0.2009 0.1957 0.1166 0.7931 0.7953 0.2111 0.2084 0.2367
2PSVM 0.4906 0.5000 0.5468 0.5006 0.4809 0.2391 0.0000 0.3871 0.3638 0.2914 0.1846 0.0000 0.3220 0.2536 0.2320 0.2689 0.0466 0.4064 0.3855 0.3193 0.6629 0.7931 0.5894 0.5975 0.5804 0.6852 0.7931 0.5919 0.6035 0.5868
DBC 0.4037 0.5724 0.6371 0.5412 0.4951 0.0044 0.0997 0.5128 0.4428 0.4572 0.0469 0.3022 0.4161 0.3524 0.3268 0.0606 0.1418 0.5335 0.4738 0.4799 0.6087 0.7562 0.5474 0.3394 0.2881 0.6723 0.7596 0.5535 0.3520 0.2991
OCSVM 0.2867 0.3124 0.5820 0.6742 0.5609 0.0808 0.1063 0.3824 0.5048 0.4155 0.0680 0.0521 0.3287 0.4967 0.3105 0.1126 0.1366 0.4067 0.5333 0.4366 0.3811 0.4434 0.7222 0.8424 0.6784 0.4742 0.5558 0.7342 0.8471 0.6879
NCM 0.5000 0.5000 0.6054 0.7339 0.6572 0.0000 0.0000 0.4418 0.6379 0.5541 0.0000 0.0000 0.3856 0.5064 0.4311 0.0466 0.0466 0.4690 0.6514 0.5720 0.7931 0.7931 0.5254 0.6641 0.5799 0.7931 0.7931 0.5399 0.6722 0.5906
PSVM 0.5000 0.2964 0.7544 0.6156 0.7333 0.0000 0.1248 0.6033 0.5041 0.6403 0.0000 0.0489 0.5689 0.3989 0.5241 0.0466 0.1526 0.6180 0.5309 0.6530 0.7931 0.4212 0.7775 0.4568 0.7274 0.7931 0.5146 0.7792 0.4662 0.7340
SVM 0.5769 0.6471 0.7295 0.7405 0.7626 0.0844 0.2098 0.5569 0.5082 0.5305 0.2741 0.4497 0.5453 0.5280 0.5698 0.1312 0.2463 0.5728 0.5327 0.5514 0.8172 0.8362 0.7793 0.7069 0.7655 0.8175 0.8368 0.7810 0.7088 0.7676
OSNN 0.5000 0.5000 0.6623 0.7781 0.7377 0.0000 0.0000 0.3441 0.6232 0.5823 0.0000 0.0000 0.4799 0.6450 0.6035 0.0466 0.0466 0.3830 0.6387 0.6038 0.7931 0.7931 0.8355 0.8514 0.8529 0.7931 0.7931 0.8355 0.8515 0.8529
SOFTMAX 0.5000 0.5000 0.5000 0.4912 0.5000 0.0000 0.0000 0.0000 0.3894 0.0000 0.0000 0.0000 0.0000 0.3340 0.0000 0.0466 0.0466 0.0466 0.4324 0.0466 0.7931 0.7931 0.7931 0.2165 0.7931 0.7931 0.7931 0.7931 0.2248 0.7931
SSVM 0.5000 0.5000 0.7247 0.7937 0.7662 0.0000 0.0000 0.5950 0.6789 0.6079 0.0000 0.0000 0.5417 0.6533 0.5633 0.0466 0.0466 0.6089 0.6909 0.6237 0.7931 0.7931 0.7841 0.8508 0.7520 0.7931 0.7931 0.7864 0.8545 0.7575
ET 0.5000 0.5000 0.7452 0.8189 0.7723 0.0000 0.0000 0.6242 0.7288 0.6712 0.0000 0.0000 0.5532 0.6306 0.6091 0.0466 0.0466 0.6359 0.7377 0.6822 0.7931 0.7931 0.7644 0.7959 0.8148 0.7931 0.7931 0.7669 0.7998 0.8155
PISVM 0.5150 0.6107 0.7419 0.8174 0.8270 0.3392 0.3990 0.6229 0.7028 0.6916 0.2703 0.3884 0.5686 0.6445 0.6410 0.3600 0.4262 0.6359 0.7132 0.7042 0.5999 0.5731 0.8009 0.8142 0.8055 0.6132 0.5857 0.8030 0.8161 0.8111
NetOpen WSVM 0.5000 0.5079 0.4760 0.4783 0.4805 0.0000 0.0748 0.3727 0.4063 0.3800 0.0000 0.0517 0.3256 0.3275 0.3270 0.0466 0.1295 0.4115 0.4382 0.4211 0.7931 0.7888 0.2011 0.2006 0.2105 0.7931 0.7953 0.2119 0.2099 0.2221
2PSVM 0.5000 0.5881 0.5468 0.5677 0.5026 0.0000 0.3615 0.3871 0.4291 0.0156 0.0000 0.3479 0.3220 0.2817 0.0150 0.0466 0.3854 0.4064 0.4593 0.0775 0.7931 0.7075 0.5894 0.7694 0.7928 0.7931 0.7130 0.5919 0.7747 0.7932
DBC 0.4691 0.4841 0.6189 0.5375 0.5138 0.1025 0.1838 0.4518 0.4797 0.4386 0.1986 0.2877 0.4005 0.3526 0.3414 0.1446 0.2221 0.4766 0.5097 0.4728 0.5863 0.4332 0.5311 0.3089 0.2701 0.6222 0.4513 0.5382 0.3223 0.2820
OCSVM 0.2867 0.3124 0.5820 0.6648 0.5623 0.0808 0.1063 0.3824 0.4840 0.4405 0.0680 0.0521 0.3287 0.4840 0.3131 0.1126 0.1366 0.4067 0.5155 0.4604 0.3811 0.4434 0.7222 0.8422 0.6775 0.4742 0.5558 0.7342 0.8441 0.6871
NCM 0.5000 0.5000 0.5579 0.7338 0.6699 0.0000 0.0000 0.2021 0.6482 0.5959 0.0000 0.0000 0.2347 0.5431 0.4631 0.0466 0.0466 0.2521 0.6592 0.6083 0.7931 0.7931 0.7963 0.7770 0.7422 0.7931 0.7931 0.7968 0.7838 0.7467
PSVM 0.4586 0.5183 0.7313 0.7237 0.6421 0.1387 0.3964 0.6079 0.5820 0.5789 0.2179 0.3069 0.5123 0.4893 0.4179 0.1780 0.4230 0.6229 0.6024 0.5899 0.5106 0.4442 0.6908 0.6200 0.6309 0.5704 0.4907 0.6968 0.6288 0.6428
SVM 0.5029 0.6272 0.7418 0.7262 0.7482 0.0326 0.1747 0.5262 0.5423 0.5809 0.0122 0.4054 0.5602 0.4991 0.5253 0.0804 0.2134 0.5455 0.5641 0.6005 0.7942 0.8375 0.7857 0.6544 0.6896 0.7943 0.8380 0.7880 0.6611 0.6964
OSNN 0.5000 0.5050 0.7407 0.7841 0.6923 0.0000 0.0591 0.5518 0.6344 0.4794 0.0000 0.0521 0.5764 0.6441 0.5384 0.0466 0.1218 0.5693 0.6485 0.5117 0.7931 0.7842 0.8120 0.8443 0.8496 0.7931 0.7843 0.8122 0.8444 0.8496
SOFTMAX 0.5000 0.5000 0.5000 0.7847 0.5000 0.0000 0.0000 0.0000 0.6276 0.0000 0.0000 0.0000 0.0000 0.6265 0.0000 0.0466 0.0466 0.0466 0.6410 0.0466 0.7931 0.7931 0.7931 0.8223 0.7931 0.7931 0.7931 0.7931 0.8224 0.7931
SSVM 0.5101 0.6825 0.7319 0.7138 0.7673 0.0283 0.3565 0.6058 0.5607 0.6133 0.0411 0.5019 0.5480 0.4768 0.5375 0.0994 0.3849 0.6191 0.5856 0.6325 0.7967 0.8101 0.7816 0.5769 0.6792 0.7967 0.8112 0.7840 0.5839 0.6852
ET 0.5729 0.6081 0.7135 0.7725 0.7787 0.2068 0.3232 0.5699 0.6357 0.6807 0.3085 0.3672 0.4824 0.5417 0.5683 0.2404 0.3506 0.5894 0.6533 0.6915 0.7448 0.7411 0.6211 0.6840 0.7394 0.7491 0.7539 0.6289 0.6913 0.7446
PISVM 0.5659 0.4891 0.7779 0.7786 0.7476 0.2647 0.3960 0.6633 0.6210 0.5805 0.3104 0.3225 0.6220 0.5660 0.5133 0.2947 0.4319 0.6754 0.6371 0.6026 0.7132 0.2655 0.8309 0.7358 0.6507 0.7157 0.2874 0.8347 0.7419 0.6596

Summary of Table 1. Each number on the fields red, orange, and yellow represents the number of times the respective color appears for each training protocol or classifier on the table above. Number into parenthesis is the sum of the numbers for every color in the same line. Each number on the field blue represents the number of times the feature, training protocol, or classifier obtains the best result per evaluation metric. Each number marked as bold represents the number of times the respective feature obtains the best result per metric and per classifier. When associated to a specific metric, number in bold represents the number of times the respective feature obtains the best result per classifier with that specific metric. Each number marked as oblique represents the number of times the respective classifier obtains the best result per training protocol and per feature. When associated to a specific training protocol, number marked as oblique represents the number of times the respective classifier obtains the best result per feature when trained with that specific training protocol. For all those numbers, when ties happen, counters are increased by 1/n, in which n is the number of ties.

Closed: 2.0000, 6.0000, 5.0000, (13.0000).
Open: 19.0000, 11.0000, 15.0000, (45.0000).
NetOpen: 9.0000, 11.0000, 10.0000, (30.0000).

WSVM: 1.0000, 1.0000, 0.0000, (2.0000).
2PSVM: 0.0000, 0.0000, 0.0000, (0.0000).
DBC: 0.0000, 0.0000, 0.0000, (0.0000).
OCSVM: 0.0000, 0.0000, 1.0000, (1.0000).
NCM: 0.0000, 0.0000, 0.0000, (0.0000).
PSVM: 0.0000, 1.0000, 1.0000, (2.0000).
SVM: 5.0000, 2.0000, 8.0000, (15.0000).
OSNN: 5.0000, 4.0000, 5.0000, (14.0000).
SOFTMAX: 0.0000, 0.0000, 0.0000, (0.0000).
SSVM: 4.0000, 8.0000, 3.0000, (15.0000).
ET: 3.0000, 5.0000, 6.5000, (14.5000).
PISVM: 12.0000, 7.0000, 5.5000, (24.5000).

rich: 0.0000.
cfa: 0.0000.
conv: 0.0000.
ip2: 4.0000.
ip1: 2.0000.

Closed: 0.0000.
Open: 6.0000.
NetOpen: 0.0000.

WSVM: 0.0000.
2PSVM: 0.0000.
DBC: 0.0000.
OCSVM: 0.0000.
NCM: 0.0000.
PSVM: 0.0000.
SVM: 0.0000.
OSNN: 1.0000.
SOFTMAX: 0.0000.
SSVM: 2.0000.
ET: 2.0000.
PISVM: 1.0000.

rich: 20.
cfa: 36.
conv: 53.
ip2: 96.
ip1: 39.

NA
 rich: 2.
 cfa: 6.
 conv: 9.
 ip2: 16.
 ip1: 9.
OSFMM
 rich: 0.
 cfa: 0.
 conv: 10.
 ip2: 20.
 ip1: 6.
OSFMm
 rich: 0.
 cfa: 2.
 conv: 11.
 ip2: 20.
 ip1: 3.
FMM
 rich: 0.
 cfa: 0.
 conv: 10.
 ip2: 20.
 ip1: 6.
FMm
 rich: 10.
 cfa: 11.
 conv: 6.
 ip2: 10.
 ip1: 6.
DA
 rich: 8.
 cfa: 17.
 conv: 7.
 ip2: 10.
 ip1: 9.

WSVM: 3.
2PSVM: 0.
DBC: 0.
OCSVM: 2.
NCM: 2.
PSVM: 5.
SVM: 17.
OSNN: 10.
SOFTMAX: 5.
SSVM: 18.
ET: 10.
PISVM: 19.

Closed
 WSVM: 3.
 2PSVM: 0.
 DBC: 0.
 OCSVM: 2.
 NCM: 0.
 PSVM: 2.
 SVM: 7.
 OSNN: 0.
 SOFTMAX: 4.
 SSVM: 12.
 ET: 0.
 PISVM: 0.
Open
 WSVM: 0.
 2PSVM: 0.
 DBC: 0.
 OCSVM: 0.
 NCM: 0.
 PSVM: 2.
 SVM: 8.
 OSNN: 5.
 SOFTMAX: 0.
 SSVM: 2.
 ET: 5.
 PISVM: 9.
NetOpen
 WSVM: 0.
 2PSVM: 0.
 DBC: 0.
 OCSVM: 0.
 NCM: 2.
 PSVM: 1.
 SVM: 2.
 OSNN: 5.
 SOFTMAX: 1.
 SSVM: 4.
 ET: 5.
 PISVM: 10.

Table 2. Open-Set Camera Model Identification (OSCMI) comparing Open and NetOpen training protocols.
Red color marks the winning training protocol for every pair of classifier and feature.
Yellow color appears when Open and NetOpen obtains the same result for the classifier-feature pair.
Metrics NA OSFMM OSFMm FMM FMm DA
Features rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1 rich cfa conv ip2 ip1
Training Protocol Classifiers
Open WSVM 0.5000 0.5079 0.4769 0.4692 0.2181 0.0000 0.0748 0.3934 0.4494 0.5236 0.0000 0.0517 0.3264 0.3215 0.1389 0.0466 0.1295 0.4288 0.4709 0.5167 0.7931 0.7888 0.2009 0.1957 0.1166 0.7931 0.7953 0.2111 0.2084 0.2367
2PSVM 0.4906 0.5000 0.5468 0.5006 0.4809 0.2391 0.0000 0.3871 0.3638 0.2914 0.1846 0.0000 0.3220 0.2536 0.2320 0.2689 0.0466 0.4064 0.3855 0.3193 0.6629 0.7931 0.5894 0.5975 0.5804 0.6852 0.7931 0.5919 0.6035 0.5868
DBC 0.4037 0.5724 0.6371 0.5412 0.4951 0.0044 0.0997 0.5128 0.4428 0.4572 0.0469 0.3022 0.4161 0.3524 0.3268 0.0606 0.1418 0.5335 0.4738 0.4799 0.6087 0.7562 0.5474 0.3394 0.2881 0.6723 0.7596 0.5535 0.3520 0.2991
OCSVM 0.2867 0.3124 0.5820 0.6742 0.5609 0.0808 0.1063 0.3824 0.5048 0.4155 0.0680 0.0521 0.3287 0.4967 0.3105 0.1126 0.1366 0.4067 0.5333 0.4366 0.3811 0.4434 0.7222 0.8424 0.6784 0.4742 0.5558 0.7342 0.8471 0.6879
NCM 0.5000 0.5000 0.6054 0.7339 0.6572 0.0000 0.0000 0.4418 0.6379 0.5541 0.0000 0.0000 0.3856 0.5064 0.4311 0.0466 0.0466 0.4690 0.6514 0.5720 0.7931 0.7931 0.5254 0.6641 0.5799 0.7931 0.7931 0.5399 0.6722 0.5906
PSVM 0.5000 0.2964 0.7544 0.6156 0.7333 0.0000 0.1248 0.6033 0.5041 0.6403 0.0000 0.0489 0.5689 0.3989 0.5241 0.0466 0.1526 0.6180 0.5309 0.6530 0.7931 0.4212 0.7775 0.4568 0.7274 0.7931 0.5146 0.7792 0.4662 0.7340
SVM 0.5769 0.6471 0.7295 0.7405 0.7626 0.0844 0.2098 0.5569 0.5082 0.5305 0.2741 0.4497 0.5453 0.5280 0.5698 0.1312 0.2463 0.5728 0.5327 0.5514 0.8172 0.8362 0.7793 0.7069 0.7655 0.8175 0.8368 0.7810 0.7088 0.7676
OSNN 0.5000 0.5000 0.6623 0.7781 0.7377 0.0000 0.0000 0.3441 0.6232 0.5823 0.0000 0.0000 0.4799 0.6450 0.6035 0.0466 0.0466 0.3830 0.6387 0.6038 0.7931 0.7931 0.8355 0.8514 0.8529 0.7931 0.7931 0.8355 0.8515 0.8529
SOFTMAX 0.5000 0.5000 0.5000 0.4912 0.5000 0.0000 0.0000 0.0000 0.3894 0.0000 0.0000 0.0000 0.0000 0.3340 0.0000 0.0466 0.0466 0.0466 0.4324 0.0466 0.7931 0.7931 0.7931 0.2165 0.7931 0.7931 0.7931 0.7931 0.2248 0.7931
SSVM 0.5000 0.5000 0.7247 0.7937 0.7662 0.0000 0.0000 0.5950 0.6789 0.6079 0.0000 0.0000 0.5417 0.6533 0.5633 0.0466 0.0466 0.6089 0.6909 0.6237 0.7931 0.7931 0.7841 0.8508 0.7520 0.7931 0.7931 0.7864 0.8545 0.7575
ET 0.5000 0.5000 0.7452 0.8189 0.7723 0.0000 0.0000 0.6242 0.7288 0.6712 0.0000 0.0000 0.5532 0.6306 0.6091 0.0466 0.0466 0.6359 0.7377 0.6822 0.7931 0.7931 0.7644 0.7959 0.8148 0.7931 0.7931 0.7669 0.7998 0.8155
PISVM 0.5150 0.6107 0.7419 0.8174 0.8270 0.3392 0.3990 0.6229 0.7028 0.6916 0.2703 0.3884 0.5686 0.6445 0.6410 0.3600 0.4262 0.6359 0.7132 0.7042 0.5999 0.5731 0.8009 0.8142 0.8055 0.6132 0.5857 0.8030 0.8161 0.8111
NetOpen WSVM 0.5000 0.5079 0.4760 0.4783 0.4805 0.0000 0.0748 0.3727 0.4063 0.3800 0.0000 0.0517 0.3256 0.3275 0.3270 0.0466 0.1295 0.4115 0.4382 0.4211 0.7931 0.7888 0.2011 0.2006 0.2105 0.7931 0.7953 0.2119 0.2099 0.2221
2PSVM 0.5000 0.5881 0.5468 0.5677 0.5026 0.0000 0.3615 0.3871 0.4291 0.0156 0.0000 0.3479 0.3220 0.2817 0.0150 0.0466 0.3854 0.4064 0.4593 0.0775 0.7931 0.7075 0.5894 0.7694 0.7928 0.7931 0.7130 0.5919 0.7747 0.7932
DBC 0.4691 0.4841 0.6189 0.5375 0.5138 0.1025 0.1838 0.4518 0.4797 0.4386 0.1986 0.2877 0.4005 0.3526 0.3414 0.1446 0.2221 0.4766 0.5097 0.4728 0.5863 0.4332 0.5311 0.3089 0.2701 0.6222 0.4513 0.5382 0.3223 0.2820
OCSVM 0.2867 0.3124 0.5820 0.6648 0.5623 0.0808 0.1063 0.3824 0.4840 0.4405 0.0680 0.0521 0.3287 0.4840 0.3131 0.1126 0.1366 0.4067 0.5155 0.4604 0.3811 0.4434 0.7222 0.8422 0.6775 0.4742 0.5558 0.7342 0.8441 0.6871
NCM 0.5000 0.5000 0.5579 0.7338 0.6699 0.0000 0.0000 0.2021 0.6482 0.5959 0.0000 0.0000 0.2347 0.5431 0.4631 0.0466 0.0466 0.2521 0.6592 0.6083 0.7931 0.7931 0.7963 0.7770 0.7422 0.7931 0.7931 0.7968 0.7838 0.7467
PSVM 0.4586 0.5183 0.7313 0.7237 0.6421 0.1387 0.3964 0.6079 0.5820 0.5789 0.2179 0.3069 0.5123 0.4893 0.4179 0.1780 0.4230 0.6229 0.6024 0.5899 0.5106 0.4442 0.6908 0.6200 0.6309 0.5704 0.4907 0.6968 0.6288 0.6428
SVM 0.5029 0.6272 0.7418 0.7262 0.7482 0.0326 0.1747 0.5262 0.5423 0.5809 0.0122 0.4054 0.5602 0.4991 0.5253 0.0804 0.2134 0.5455 0.5641 0.6005 0.7942 0.8375 0.7857 0.6544 0.6896 0.7943 0.8380 0.7880 0.6611 0.6964
OSNN 0.5000 0.5050 0.7407 0.7841 0.6923 0.0000 0.0591 0.5518 0.6344 0.4794 0.0000 0.0521 0.5764 0.6441 0.5384 0.0466 0.1218 0.5693 0.6485 0.5117 0.7931 0.7842 0.8120 0.8443 0.8496 0.7931 0.7843 0.8122 0.8444 0.8496
SOFTMAX 0.5000 0.5000 0.5000 0.7847 0.5000 0.0000 0.0000 0.0000 0.6276 0.0000 0.0000 0.0000 0.0000 0.6265 0.0000 0.0466 0.0466 0.0466 0.6410 0.0466 0.7931 0.7931 0.7931 0.8223 0.7931 0.7931 0.7931 0.7931 0.8224 0.7931
SSVM 0.5101 0.6825 0.7319 0.7138 0.7673 0.0283 0.3565 0.6058 0.5607 0.6133 0.0411 0.5019 0.5480 0.4768 0.5375 0.0994 0.3849 0.6191 0.5856 0.6325 0.7967 0.8101 0.7816 0.5769 0.6792 0.7967 0.8112 0.7840 0.5839 0.6852
ET 0.5729 0.6081 0.7135 0.7725 0.7787 0.2068 0.3232 0.5699 0.6357 0.6807 0.3085 0.3672 0.4824 0.5417 0.5683 0.2404 0.3506 0.5894 0.6533 0.6915 0.7448 0.7411 0.6211 0.6840 0.7394 0.7491 0.7539 0.6289 0.6913 0.7446
PISVM 0.5659 0.4891 0.7779 0.7786 0.7476 0.2647 0.3960 0.6633 0.6210 0.5805 0.3104 0.3225 0.6220 0.5660 0.5133 0.2947 0.4319 0.6754 0.6371 0.6026 0.7132 0.2655 0.8309 0.7358 0.6507 0.7157 0.2874 0.8347 0.7419 0.6596

Summary of Table 2. The fractional number shows the mean of the differences in accuracy for the cases in which the corresponding training protocol obtains better accuracy than competing training protocol. Number into parenthesis counts the number of times the corresponding training protocol obtains better result than competing training protocol. When ties happen, number into parenthesis is increased by 0.5. When associated to a specific feature, classifier, or metric, the same information mentioned before are obtained limited to that feature, classifier, or metric, respectively.

Open: 0.0771 (184).
NetOpen: 0.1054 (176).

rich
 Open: 0.1101 (33).
 NetOpen: 0.0985 (39).
cfa
 Open: 0.0981 (32).
 NetOpen: 0.1808 (40).
conv
 Open: 0.0574 (39).
 NetOpen: 0.0589 (33).
ip2
 Open: 0.0634 (37).
 NetOpen: 0.1130 (35).
ip1
 Open: 0.0791 (43).
 NetOpen: 0.0635 (29).

WSVM
 Open: 0.0410 (15).
 NetOpen: 0.0630 (15).
2PSVM
 Open: 0.1933 (11).
 NetOpen: 0.1501 (19).
DBC
 Open: 0.0587 (19).
 NetOpen: 0.0609 (11).
OCSVM
 Open: 0.0082 (17).
 NetOpen: 0.0132 (13).
NCM
 Open: 0.1310 (11).
 NetOpen: 0.0960 (19).
PSVM
 Open: 0.0949 (14).
 NetOpen: 0.1385 (16).
SVM
 Open: 0.0512 (20).
 NetOpen: 0.0208 (10).
OSNN
 Open: 0.0301 (16).
 NetOpen: 0.0716 (14).
SOFTMAX
 Open: nan (12).
 NetOpen: 0.3727 (18).
SSVM
 Open: 0.1091 (11).
 NetOpen: 0.0844 (19).
ET
 Open: 0.0731 (19).
 NetOpen: 0.1736 (11).
PISVM
 Open: 0.1100 (19).
 NetOpen: 0.0494 (11).

NA
 Open: 0.0424 (27.5).
 NetOpen: 0.0676 (32.5).
OSFMM
 Open: 0.0895 (27.5).
 NetOpen: 0.1076 (32.5).
OSFMm
 Open: 0.0817 (29.5).
 NetOpen: 0.1312 (30.5).
FMM
 Open: 0.0827 (26.5).
 NetOpen: 0.0993 (33.5).
FMm
 Open: 0.0854 (35.5).
 NetOpen: 0.1180 (24.5).
DA
 Open: 0.0772 (37.5).
 NetOpen: 0.1210 (22.5).

Table 3. Open-Set Camera Model Identification (OSCMI) results for WSVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.371419 0.332065 0.254028 0.362347 0.156330 0.209641
cfa 0.448061 0.345541 0.305268 0.390653 0.193422 0.217675
conv 0.475964 0.372746 0.325578 0.411533 0.201081 0.211894
ip2 0.478316 0.406303 0.327527 0.438229 0.200556 0.209866
ip1 0.480452 0.379956 0.327025 0.421054 0.210542 0.222105
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.507853 0.074833 0.051668 0.129465 0.788782 0.795315
conv 0.476942 0.393357 0.326375 0.428810 0.200931 0.211143
ip2 0.469202 0.449390 0.321507 0.470888 0.195675 0.208365
ip1 0.218128 0.523623 0.138918 0.516659 0.116609 0.236672
NetOpen rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.507853 0.074833 0.051668 0.129465 0.788782 0.795315
conv 0.475964 0.372746 0.325578 0.411533 0.201081 0.211894
ip2 0.478316 0.406303 0.327527 0.438229 0.200556 0.209866
ip1 0.480452 0.379956 0.327025 0.421054 0.210542 0.222105

Table 4. Open-Set Camera Model Identification (OSCMI) results for 2PSVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.485457 0.233864 0.180174 0.263437 0.655579 0.678781
cfa 0.588059 0.361536 0.347898 0.385358 0.707464 0.713020
conv 0.537069 0.379165 0.306571 0.399694 0.596786 0.600841
ip2 0.411246 0.289515 0.213883 0.315736 0.430395 0.436477
ip1 0.458891 0.294454 0.222748 0.320901 0.537468 0.543926
Open rich 0.490585 0.239087 0.184605 0.268938 0.662862 0.685238
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.546784 0.387092 0.321952 0.406441 0.589428 0.591906
ip2 0.500650 0.363762 0.253602 0.385475 0.597537 0.603469
ip1 0.480869 0.291372 0.232029 0.319303 0.580417 0.586800
NetOpen rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.588059 0.361536 0.347898 0.385358 0.707464 0.713020
conv 0.546784 0.387092 0.321952 0.406441 0.589428 0.591906
ip2 0.567680 0.429093 0.281664 0.459269 0.769410 0.774741
ip1 0.502628 0.015554 0.014968 0.077456 0.792837 0.793212

Table 5. Open-Set Camera Model Identification (OSCMI) results for DBC classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.468334 0.143113 0.211895 0.181764 0.555639 0.602793
cfa 0.344743 0.216283 0.215623 0.253520 0.218501 0.279922
conv 0.554519 0.447526 0.361354 0.476670 0.355909 0.364995
ip2 0.493597 0.413913 0.333653 0.452631 0.227136 0.237648
ip1 0.499095 0.405190 0.335869 0.445477 0.237348 0.248235
Open rich 0.403734 0.004412 0.046885 0.060601 0.608725 0.672323
cfa 0.572362 0.099699 0.302247 0.141848 0.756195 0.759574
conv 0.637124 0.512768 0.416093 0.533470 0.547379 0.553537
ip2 0.541154 0.442787 0.352355 0.473760 0.339390 0.352005
ip1 0.495140 0.457189 0.326807 0.479899 0.288106 0.299144
NetOpen rich 0.469134 0.102547 0.198608 0.144628 0.586274 0.622165
cfa 0.484146 0.183835 0.287726 0.222137 0.433248 0.451269
conv 0.618937 0.451842 0.400530 0.476624 0.531086 0.538219
ip2 0.537498 0.479693 0.352584 0.509663 0.308905 0.322346
ip1 0.513830 0.438565 0.341421 0.472809 0.270086 0.282024

Table 6. Open-Set Camera Model Identification (OSCMI) results for OCSVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.163128 0.010434 0.016397 0.060243 0.237273 0.384367
cfa 0.316658 0.120234 0.056700 0.149517 0.445487 0.558042
conv 0.570533 0.390828 0.312676 0.413981 0.703709 0.719703
ip2 0.646720 0.481112 0.437583 0.508252 0.816264 0.822646
ip1 0.547836 0.417515 0.291996 0.439135 0.668344 0.680057
Open rich 0.286690 0.080819 0.068023 0.112574 0.381138 0.474170
cfa 0.312381 0.106297 0.052128 0.136618 0.443385 0.555789
conv 0.582043 0.382428 0.328696 0.406683 0.722181 0.734194
ip2 0.674193 0.504835 0.496741 0.533305 0.842394 0.847124
ip1 0.560886 0.415527 0.310507 0.436606 0.678405 0.687941
NetOpen rich 0.286690 0.080819 0.068023 0.112574 0.381138 0.474170
cfa 0.312381 0.106297 0.052128 0.136618 0.443385 0.555789
conv 0.582043 0.382428 0.328696 0.406683 0.722181 0.734194
ip2 0.664795 0.483990 0.483996 0.515548 0.842168 0.844121
ip1 0.562330 0.440492 0.313092 0.460377 0.677504 0.687115

Table 7. Open-Set Camera Model Identification (OSCMI) results for NCM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.078947 0.137298 0.054128 0.130072 0.032663 0.206863
cfa 0.092377 0.165919 0.063336 0.157186 0.038219 0.206863
conv 0.447187 0.349903 0.306601 0.331487 0.185013 0.206863
ip2 0.473321 0.367603 0.324519 0.348256 0.195825 0.206863
ip1 0.457169 0.367501 0.313445 0.348158 0.189143 0.206863
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.605412 0.441829 0.385551 0.469035 0.525379 0.539946
ip2 0.733938 0.637938 0.506411 0.651430 0.664139 0.672248
ip1 0.657217 0.554137 0.431135 0.571970 0.579892 0.590554
NetOpen rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.557930 0.202082 0.234665 0.252115 0.796291 0.796816
ip2 0.733848 0.648229 0.543147 0.659164 0.776994 0.783751
ip1 0.669942 0.595866 0.463100 0.608329 0.742229 0.746659

Table 8. Open-Set Camera Model Identification (OSCMI) results for PSVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.248970 0.275517 0.170546 0.305039 0.103394 0.207388
cfa 0.387822 0.386458 0.262318 0.419437 0.172774 0.223532
conv 0.473565 0.435344 0.323441 0.471369 0.201532 0.214447
ip2 0.477913 0.400158 0.327329 0.441669 0.199279 0.208965
ip1 0.476903 0.411031 0.326148 0.450904 0.201081 0.211969
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.296357 0.124776 0.048869 0.152621 0.421159 0.514567
conv 0.754436 0.603296 0.568871 0.617996 0.777519 0.779246
ip2 0.615588 0.504061 0.398922 0.530926 0.456825 0.466211
ip1 0.733280 0.640315 0.524053 0.653044 0.727361 0.734044
NetOpen rich 0.458580 0.138696 0.217898 0.178022 0.510587 0.570431
cfa 0.518290 0.396372 0.306921 0.423039 0.444211 0.490689
conv 0.731290 0.607867 0.512314 0.622858 0.690794 0.696801
ip2 0.723679 0.582003 0.489301 0.602382 0.619988 0.628848
ip1 0.642136 0.578884 0.417938 0.589936 0.630876 0.642814

Table 9. Open-Set Camera Model Identification (OSCMI) results for SVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.502943 0.032567 0.012248 0.080422 0.794188 0.794263
cfa 0.627248 0.174661 0.405368 0.213432 0.837513 0.838039
conv 0.762147 0.552360 0.573147 0.570889 0.772939 0.775041
ip2 0.701231 0.544466 0.465287 0.570739 0.559694 0.567578
ip1 0.761249 0.597969 0.530751 0.618287 0.674050 0.680958
Open rich 0.576905 0.084425 0.274054 0.131155 0.817240 0.817465
cfa 0.647103 0.209777 0.449698 0.246349 0.836237 0.836762
conv 0.729546 0.556918 0.545287 0.572832 0.779321 0.781048
ip2 0.740481 0.508152 0.527985 0.532726 0.706863 0.708815
ip1 0.762558 0.530523 0.569829 0.551414 0.765505 0.767608
NetOpen rich 0.502943 0.032567 0.012248 0.080422 0.794188 0.794263
cfa 0.627248 0.174661 0.405368 0.213432 0.837513 0.838039
conv 0.741753 0.526234 0.560232 0.545453 0.785704 0.787956
ip2 0.726175 0.542280 0.499146 0.564054 0.654378 0.661060
ip1 0.748241 0.580894 0.525287 0.600475 0.689593 0.696351

Table 10. Open-Set Camera Model Identification (OSCMI) results for OSNN classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.252545 0.215743 0.173118 0.244396 0.104595 0.206938
cfa 0.288203 0.283738 0.197598 0.268804 0.119237 0.206863
conv 0.475318 0.344115 0.325888 0.326003 0.196651 0.206863
ip2 0.479886 0.396778 0.328552 0.438681 0.200706 0.209791
ip1 0.475554 0.362910 0.325709 0.409143 0.198303 0.208965
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.662326 0.344061 0.479943 0.383030 0.835486 0.835486
ip2 0.778074 0.623170 0.645034 0.638663 0.851404 0.851479
ip1 0.737701 0.582294 0.603522 0.603828 0.852906 0.852906
NetOpen rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.504965 0.059061 0.052094 0.121841 0.784202 0.784277
conv 0.740746 0.551789 0.576449 0.569306 0.811984 0.812209
ip2 0.784088 0.634361 0.644132 0.648524 0.844346 0.844421
ip1 0.692287 0.479434 0.538373 0.511713 0.849602 0.849602

Table 11. Open-Set Camera Model Identification (OSCMI) results for SOFTMAX classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ip2 0.479492 0.383314 0.328750 0.363139 0.198378 0.206863
ip1 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ip2 0.491168 0.389397 0.333965 0.432442 0.216474 0.224809
ip1 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
NetOpen rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ip2 0.784707 0.627554 0.626479 0.641033 0.822346 0.822421
ip1 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137

Table 12. Open-Set Camera Model Identification (OSCMI) results for SSVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.535343 0.097937 0.137496 0.168530 0.805151 0.805301
conv 0.762981 0.638464 0.578863 0.650939 0.786605 0.790734
ip2 0.776418 0.621341 0.551494 0.639865 0.705136 0.712419
ip1 0.799334 0.627454 0.590385 0.645341 0.755744 0.762277
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.724661 0.594981 0.541726 0.608934 0.784127 0.786379
ip2 0.793659 0.678869 0.653270 0.690859 0.850803 0.854483
ip1 0.766246 0.607892 0.563300 0.623721 0.751990 0.757546
NetOpen rich 0.510053 0.028293 0.041091 0.099405 0.796741 0.796741
cfa 0.682548 0.356467 0.501861 0.384892 0.810107 0.811158
conv 0.731905 0.605817 0.548025 0.619149 0.781574 0.783977
ip2 0.713815 0.560661 0.476756 0.585588 0.576888 0.583947
ip1 0.767332 0.613322 0.537472 0.632472 0.679231 0.685163

Table 13. Open-Set Camera Model Identification (OSCMI) results for ET classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.320508 0.252201 0.219747 0.238928 0.132602 0.206863
cfa 0.373684 0.281375 0.256206 0.266566 0.154603 0.206863
conv 0.471688 0.365204 0.323399 0.345983 0.195149 0.206863
ip2 0.480517 0.398881 0.328733 0.440479 0.202132 0.211368
ip1 0.475681 0.388471 0.326137 0.368025 0.196801 0.206863
Open rich 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
cfa 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
conv 0.745213 0.624245 0.553191 0.635916 0.764379 0.766857
ip2 0.818889 0.728814 0.630568 0.737745 0.795915 0.799820
ip1 0.772327 0.671218 0.609066 0.682170 0.814837 0.815513
NetOpen rich 0.572948 0.206752 0.308523 0.240384 0.744781 0.749061
cfa 0.608058 0.323159 0.367197 0.350592 0.741102 0.753942
conv 0.713523 0.569853 0.482412 0.589437 0.621114 0.628923
ip2 0.772508 0.635655 0.541680 0.653289 0.684037 0.691320
ip1 0.778685 0.680659 0.568293 0.691518 0.739375 0.744556

Table 14. Open-Set Camera Model Identification (OSCMI) results for PISVM classifier.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 5 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Metrics
Closed rich 0.354446 0.325550 0.243016 0.308416 0.146644 0.206863
cfa 0.437750 0.383759 0.300131 0.363561 0.181108 0.206863
conv 0.470599 0.422993 0.322653 0.400731 0.194699 0.206863
ip2 0.475318 0.418334 0.325888 0.396316 0.196651 0.206863
ip1 0.471688 0.391182 0.323399 0.370594 0.195149 0.206863
Open rich 0.514996 0.339185 0.270342 0.360027 0.599865 0.613155
cfa 0.610652 0.398973 0.388384 0.426217 0.573059 0.585749
conv 0.741924 0.622852 0.568624 0.635945 0.800871 0.803049
ip2 0.817379 0.702806 0.644492 0.713177 0.814161 0.816114
ip1 0.827047 0.691605 0.640991 0.704250 0.805451 0.811083
NetOpen rich 0.565946 0.264684 0.310419 0.294713 0.713245 0.715723
cfa 0.489076 0.395985 0.322529 0.431923 0.265505 0.287431
conv 0.777896 0.663267 0.622003 0.675372 0.830906 0.834660
ip2 0.778607 0.620982 0.565983 0.637066 0.735846 0.741853
ip1 0.747601 0.580532 0.513265 0.602591 0.650698 0.659558

Table 15. Open-Set Camera Model Identification (OSCMI) results for rich feature.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 12 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Classifiers
Closed WSVM 0.371419 0.332065 0.254028 0.362347 0.156330 0.209641
2PSVM 0.485457 0.233864 0.180174 0.263437 0.655579 0.678781
DBC 0.468334 0.143113 0.211895 0.181764 0.555639 0.602793
OCSVM 0.163128 0.010434 0.016397 0.060243 0.237273 0.384367
NCM 0.078947 0.137298 0.054128 0.130072 0.032663 0.206863
PSVM 0.248970 0.275517 0.170546 0.305039 0.103394 0.207388
SVM 0.502943 0.032567 0.012248 0.080422 0.794188 0.794263
OSNN 0.252545 0.215743 0.173118 0.244396 0.104595 0.206938
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ET 0.320508 0.252201 0.219747 0.238928 0.132602 0.206863
PISVM 0.354446 0.325550 0.243016 0.308416 0.146644 0.206863
Open WSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
2PSVM 0.490585 0.239087 0.184605 0.268938 0.662862 0.685238
DBC 0.403734 0.004412 0.046885 0.060601 0.608725 0.672323
OCSVM 0.286690 0.080819 0.068023 0.112574 0.381138 0.474170
NCM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SVM 0.576905 0.084425 0.274054 0.131155 0.817240 0.817465
OSNN 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ET 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PISVM 0.514996 0.339185 0.270342 0.360027 0.599865 0.613155
NetOpen WSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
2PSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
DBC 0.469134 0.102547 0.198608 0.144628 0.586274 0.622165
OCSVM 0.286690 0.080819 0.068023 0.112574 0.381138 0.474170
NCM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PSVM 0.458580 0.138696 0.217898 0.178022 0.510587 0.570431
SVM 0.502943 0.032567 0.012248 0.080422 0.794188 0.794263
OSNN 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.510053 0.028293 0.041091 0.099405 0.796741 0.796741
ET 0.572948 0.206752 0.308523 0.240384 0.744781 0.749061
PISVM 0.565946 0.264684 0.310419 0.294713 0.713245 0.715723

Table 16. Open-Set Camera Model Identification (OSCMI) results for cfa feature.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 12 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Classifiers
Closed WSVM 0.448061 0.345541 0.305268 0.390653 0.193422 0.217675
2PSVM 0.588059 0.361536 0.347898 0.385358 0.707464 0.713020
DBC 0.344743 0.216283 0.215623 0.253520 0.218501 0.279922
OCSVM 0.316658 0.120234 0.056700 0.149517 0.445487 0.558042
NCM 0.092377 0.165919 0.063336 0.157186 0.038219 0.206863
PSVM 0.387822 0.386458 0.262318 0.419437 0.172774 0.223532
SVM 0.627248 0.174661 0.405368 0.213432 0.837513 0.838039
OSNN 0.288203 0.283738 0.197598 0.268804 0.119237 0.206863
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.535343 0.097937 0.137496 0.168530 0.805151 0.805301
ET 0.373684 0.281375 0.256206 0.266566 0.154603 0.206863
PISVM 0.437750 0.383759 0.300131 0.363561 0.181108 0.206863
Open WSVM 0.507853 0.074833 0.051668 0.129465 0.788782 0.795315
2PSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
DBC 0.572362 0.099699 0.302247 0.141848 0.756195 0.759574
OCSVM 0.312381 0.106297 0.052128 0.136618 0.443385 0.555789
NCM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PSVM 0.296357 0.124776 0.048869 0.152621 0.421159 0.514567
SVM 0.647103 0.209777 0.449698 0.246349 0.836237 0.836762
OSNN 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
ET 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PISVM 0.610652 0.398973 0.388384 0.426217 0.573059 0.585749
NetOpen WSVM 0.507853 0.074833 0.051668 0.129465 0.788782 0.795315
2PSVM 0.588059 0.361536 0.347898 0.385358 0.707464 0.713020
DBC 0.484146 0.183835 0.287726 0.222137 0.433248 0.451269
OCSVM 0.312381 0.106297 0.052128 0.136618 0.443385 0.555789
NCM 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
PSVM 0.518290 0.396372 0.306921 0.423039 0.444211 0.490689
SVM 0.627248 0.174661 0.405368 0.213432 0.837513 0.838039
OSNN 0.504965 0.059061 0.052094 0.121841 0.784202 0.784277
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.682548 0.356467 0.501861 0.384892 0.810107 0.811158
ET 0.608058 0.323159 0.367197 0.350592 0.741102 0.753942
PISVM 0.489076 0.395985 0.322529 0.431923 0.265505 0.287431

Table 17. Open-Set Camera Model Identification (OSCMI) results for conv feature.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 12 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Classifiers
Closed WSVM 0.475964 0.372746 0.325578 0.411533 0.201081 0.211894
2PSVM 0.537069 0.379165 0.306571 0.399694 0.596786 0.600841
DBC 0.554519 0.447526 0.361354 0.476670 0.355909 0.364995
OCSVM 0.570533 0.390828 0.312676 0.413981 0.703709 0.719703
NCM 0.447187 0.349903 0.306601 0.331487 0.185013 0.206863
PSVM 0.473565 0.435344 0.323441 0.471369 0.201532 0.214447
SVM 0.762147 0.552360 0.573147 0.570889 0.772939 0.775041
OSNN 0.475318 0.344115 0.325888 0.326003 0.196651 0.206863
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.762981 0.638464 0.578863 0.650939 0.786605 0.790734
ET 0.471688 0.365204 0.323399 0.345983 0.195149 0.206863
PISVM 0.470599 0.422993 0.322653 0.400731 0.194699 0.206863
Open WSVM 0.476942 0.393357 0.326375 0.428810 0.200931 0.211143
2PSVM 0.546784 0.387092 0.321952 0.406441 0.589428 0.591906
DBC 0.637124 0.512768 0.416093 0.533470 0.547379 0.553537
OCSVM 0.582043 0.382428 0.328696 0.406683 0.722181 0.734194
NCM 0.605412 0.441829 0.385551 0.469035 0.525379 0.539946
PSVM 0.754436 0.603296 0.568871 0.617996 0.777519 0.779246
SVM 0.729546 0.556918 0.545287 0.572832 0.779321 0.781048
OSNN 0.662326 0.344061 0.479943 0.383030 0.835486 0.835486
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.724661 0.594981 0.541726 0.608934 0.784127 0.786379
ET 0.745213 0.624245 0.553191 0.635916 0.764379 0.766857
PISVM 0.741924 0.622852 0.568624 0.635945 0.800871 0.803049
NetOpen WSVM 0.475964 0.372746 0.325578 0.411533 0.201081 0.211894
2PSVM 0.546784 0.387092 0.321952 0.406441 0.589428 0.591906
DBC 0.618937 0.451842 0.400530 0.476624 0.531086 0.538219
OCSVM 0.582043 0.382428 0.328696 0.406683 0.722181 0.734194
NCM 0.557930 0.202082 0.234665 0.252115 0.796291 0.796816
PSVM 0.731290 0.607867 0.512314 0.622858 0.690794 0.696801
SVM 0.741753 0.526234 0.560232 0.545453 0.785704 0.787956
OSNN 0.740746 0.551789 0.576449 0.569306 0.811984 0.812209
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.731905 0.605817 0.548025 0.619149 0.781574 0.783977
ET 0.713523 0.569853 0.482412 0.589437 0.621114 0.628923
PISVM 0.777896 0.663267 0.622003 0.675372 0.830906 0.834660

Table 18. Open-Set Camera Model Identification (OSCMI) results for ip2 feature.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 12 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Classifiers
Closed WSVM 0.478316 0.406303 0.327527 0.438229 0.200556 0.209866
2PSVM 0.411246 0.289515 0.213883 0.315736 0.430395 0.436477
DBC 0.493597 0.413913 0.333653 0.452631 0.227136 0.237648
OCSVM 0.646720 0.481112 0.437583 0.508252 0.816264 0.822646
NCM 0.473321 0.367603 0.324519 0.348256 0.195825 0.206863
PSVM 0.477913 0.400158 0.327329 0.441669 0.199279 0.208965
SVM 0.701231 0.544466 0.465287 0.570739 0.559694 0.567578
OSNN 0.479886 0.396778 0.328552 0.438681 0.200706 0.209791
SOFTMAX 0.479492 0.383314 0.328750 0.363139 0.198378 0.206863
SSVM 0.776418 0.621341 0.551494 0.639865 0.705136 0.712419
ET 0.480517 0.398881 0.328733 0.440479 0.202132 0.211368
PISVM 0.475318 0.418334 0.325888 0.396316 0.196651 0.206863
Open WSVM 0.469202 0.449390 0.321507 0.470888 0.195675 0.208365
2PSVM 0.500650 0.363762 0.253602 0.385475 0.597537 0.603469
DBC 0.541154 0.442787 0.352355 0.473760 0.339390 0.352005
OCSVM 0.674193 0.504835 0.496741 0.533305 0.842394 0.847124
NCM 0.733938 0.637938 0.506411 0.651430 0.664139 0.672248
PSVM 0.615588 0.504061 0.398922 0.530926 0.456825 0.466211
SVM 0.740481 0.508152 0.527985 0.532726 0.706863 0.708815
OSNN 0.778074 0.623170 0.645034 0.638663 0.851404 0.851479
SOFTMAX 0.491168 0.389397 0.333965 0.432442 0.216474 0.224809
SSVM 0.793659 0.678869 0.653270 0.690859 0.850803 0.854483
ET 0.818889 0.728814 0.630568 0.737745 0.795915 0.799820
PISVM 0.817379 0.702806 0.644492 0.713177 0.814161 0.816114
NetOpen WSVM 0.478316 0.406303 0.327527 0.438229 0.200556 0.209866
2PSVM 0.567680 0.429093 0.281664 0.459269 0.769410 0.774741
DBC 0.537498 0.479693 0.352584 0.509663 0.308905 0.322346
OCSVM 0.664795 0.483990 0.483996 0.515548 0.842168 0.844121
NCM 0.733848 0.648229 0.543147 0.659164 0.776994 0.783751
PSVM 0.723679 0.582003 0.489301 0.602382 0.619988 0.628848
SVM 0.726175 0.542280 0.499146 0.564054 0.654378 0.661060
OSNN 0.784088 0.634361 0.644132 0.648524 0.844346 0.844421
SOFTMAX 0.784707 0.627554 0.626479 0.641033 0.822346 0.822421
SSVM 0.713815 0.560661 0.476756 0.585588 0.576888 0.583947
ET 0.772508 0.635655 0.541680 0.653289 0.684037 0.691320
PISVM 0.778607 0.620982 0.565983 0.637066 0.735846 0.741853

Table 19. Open-Set Camera Model Identification (OSCMI) results for ip1 feature.
Red, orange, and yellow mark the 3 best results per metric (column), respectively.
Oblique indicates the best result per training protocol (group of 12 rows) and per metric (column).
NA OSFMM OSFMm FMM FMm DA
Training Protocol Classifiers
Closed WSVM 0.480452 0.379956 0.327025 0.421054 0.210542 0.222105
2PSVM 0.458891 0.294454 0.222748 0.320901 0.537468 0.543926
DBC 0.499095 0.405190 0.335869 0.445477 0.237348 0.248235
OCSVM 0.547836 0.417515 0.291996 0.439135 0.668344 0.680057
NCM 0.457169 0.367501 0.313445 0.348158 0.189143 0.206863
PSVM 0.476903 0.411031 0.326148 0.450904 0.201081 0.211969
SVM 0.761249 0.597969 0.530751 0.618287 0.674050 0.680958
OSNN 0.475554 0.362910 0.325709 0.409143 0.198303 0.208965
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.799334 0.627454 0.590385 0.645341 0.755744 0.762277
ET 0.475681 0.388471 0.326137 0.368025 0.196801 0.206863
PISVM 0.471688 0.391182 0.323399 0.370594 0.195149 0.206863
Open WSVM 0.218128 0.523623 0.138918 0.516659 0.116609 0.236672
2PSVM 0.480869 0.291372 0.232029 0.319303 0.580417 0.586800
DBC 0.495140 0.457189 0.326807 0.479899 0.288106 0.299144
OCSVM 0.560886 0.415527 0.310507 0.436606 0.678405 0.687941
NCM 0.657217 0.554137 0.431135 0.571970 0.579892 0.590554
PSVM 0.733280 0.640315 0.524053 0.653044 0.727361 0.734044
SVM 0.762558 0.530523 0.569829 0.551414 0.765505 0.767608
OSNN 0.737701 0.582294 0.603522 0.603828 0.852906 0.852906
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.766246 0.607892 0.563300 0.623721 0.751990 0.757546
ET 0.772327 0.671218 0.609066 0.682170 0.814837 0.815513
PISVM 0.827047 0.691605 0.640991 0.704250 0.805451 0.811083
NetOpen WSVM 0.480452 0.379956 0.327025 0.421054 0.210542 0.222105
2PSVM 0.502628 0.015554 0.014968 0.077456 0.792837 0.793212
DBC 0.513830 0.438565 0.341421 0.472809 0.270086 0.282024
OCSVM 0.562330 0.440492 0.313092 0.460377 0.677504 0.687115
NCM 0.669942 0.595866 0.463100 0.608329 0.742229 0.746659
PSVM 0.642136 0.578884 0.417938 0.589936 0.630876 0.642814
SVM 0.748241 0.580894 0.525287 0.600475 0.689593 0.696351
OSNN 0.692287 0.479434 0.538373 0.511713 0.849602 0.849602
SOFTMAX 0.500000 0.000000 0.000000 0.046560 0.793137 0.793137
SSVM 0.767332 0.613322 0.537472 0.632472 0.679231 0.685163
ET 0.778685 0.680659 0.568293 0.691518 0.739375 0.744556
PISVM 0.747601 0.580532 0.513265 0.602591 0.650698 0.659558

Raw results for all combinations of features, classifiers, and training protocol as well as for all metrics, is available here as a Python dictionary. Each key of the dictionary is in the form (protocol, classifier, feature, metric). The value of each corresponding key represents the accuracy for the corresponding metric.

Last updated on June 02, 2019.