Fractional Rectified Linear Unit Activation Function and Its Variants

Table 4

Performance comparison of neural network model in Case 3 with various activation functions.

Hidden Layer 1	Hidden Layer 2	Training		Testing
			MSE		MSE
ReLU	ReLU	0.9058	0.0991	0.9203	0.0971
	LReLU	0.9504	0.0471	0.9569	0.0544
	PReLU	0.9889	0.0119	0.9881	0.0122
	ELU	0.9547	0.0365	0.9536	0.0426
	SiLU	0.9405	0.0613	0.9373	0.0641
	GELU	0.9554	0.0482	0.9579	0.0451
	FReLU	0.9065	0.1384	0.8879	0.1462
	FLReLU	0.8922	0.0884	0.8757	0.0907
	FPReLU	0.9826	0.0148	0.9788	0.0189
	FELU	0.8292	0.1481	0.8175	0.1851
	FSiLU	0.7779	0.1824	0.7881	0.1811
	FGELU	0.6771	1.7891	0.6654	1.6541
LReLU	ReLU	0.9569	0.0331	0.9601	0.0319
	LReLU	0.9363	0.0484	0.9297	0.0484
	PReLU	0.9157	0.0812	0.9084	0.0871
	ELU	0.9357	0.0694	0.9308	0.0819
	SiLU	0.9355	0.0617	0.9399	0.0602
	GELU	0.9313	0.0571	0.9267	0.0567
	FReLU	0.9461	0.0404	0.9488	0.0391
	FLReLU	0.9464	0.0435	0.9493	0.0433
	FPReLU	0.9592	0.0312	0.9579	0.0213
	FELU	0.8629	0.1687	0.8633	0.1143
	FSiLU	0.6889	0.6168	0.5408	0.6224
	FGELU	0.5111	5.3639	0.5137	5.2435
PReLU	ReLU	0.9261	0.0106	0.9396	0.0109
	LReLU	0.9457	0.0425	0.9467	0.0404
	PReLU	0.9363	0.0672	0.9266	0.0584
	ELU	0.9568	0.0666	0.9519	0.0701
	SiLU	0.9283	0.0922	0.9333	0.0911
	GELU	0.9547	0.0348	0.9687	0.0271
	FReLU	0.9553	0.0352	0.9588	0.0331
	FLReLU	0.9668	0.0261	0.9653	0.0256
	FPReLU	0.9339	0.0966	0.9349	0.0887
	FELU	0.8724	0.1473	0.8807	0.2205
	FSiLU	0.8511	0.1251	0.8423	0.1281
	FGELU	0.4115	4.3223	0.4112	4.1112
ELU	ReLU	0.9321	0.0791	0.9363	0.0745
	LReLU	0.9301	0.5374	0.9348	0.0512
	PReLU	0.9834	0.0144	0.9829	0.0143
	ELU	0.8773	0.1551	0.8686	0.1333
	SiLU	0.9383	0.0932	0.9356	0.0921
	GELU	0.9157	0.1554	0.9105	0.1543
	FReLU	0.8568	0.1497	0.8529	0.1433
	FLReLU	0.9288	0.0769	0.9105	0.0823
	FPReLU	0.9096	0.0657	0.9166	0.0633
	FELU	0.8412	0.1737	0.8512	0.1612
	FSiLU	0.4697	0.6491	0.5604	0.6543
	FGELU	0.3072	4.1245	0.3786	4.4532
SiLU	ReLU	0.9236	0.0621	0.9194	0.0681
	LReLU	0.9576	0.0322	0.9579	0.0312
	PReLU	0.0979	0.0151	0.9745	0.0212
	ELU	0.9172	0.1121	0.9171	0.1221
	SiLU	0.8771	0.0943	0.8543	0.1177
	GELU	0.9304	0.0682	0.9281	0.0636
	FReLU	0.8411	0.1321	0.8441	0.1285
	FLReLU	0.8608	0.1111	0.8441	0.1153
	FPReLU	0.9789	0.0166	0.9755	0.0167
	FELU	0.8773	0.4941	0.8758	0.4962
	FSiLU	0.8001	0.3682	0.8126	0.3214
	FGELU	0.6267	1.9871	0.6691	1.9887
GELU	ReLU	0.8796	0.1413	0.8721	0.1383
	LReLU	0.9172	0.0629	0.9388	0.0431
	PReLU	0.9152	0.2983	0.8981	0.3958
	ELU	0.9152	0.0631	0.9162	0.0594
	SiLU	0.9548	0.0356	0.9605	0.3114
	GELU	0.9324	0.0667	0.9317	0.0637
	FReLU	0.9068	0.0731	0.8999	0.0777
	FLReLU	0.9253	0.0615	0.9304	0.0641
	FPReLU	0.9551	0.0371	0.9497	0.0358
	FELU	0.9068	0.0794	0.9144	0.0706
	FSiLU	0.8103	0.4528	0.8243	0.5552
	FGELU	0.6089	1.8838	0.6226	1.8921
FReLU	ReLU	0.9701	0.0236	0.9652	0.0236
	LReLU	0.9887	0.0084	0.9884	0.0092
	PReLU	0.9843	0.0118	0.9801	0.0157
	ELU	0.9837	0.0131	0.9845	0.0113
	SiLU	0.9602	0.0334	0.9621	0.0285
	GELU	0.9581	0.0321	0.9541	0.0353
	FReLU	0.9084	0.1041	0.9081	0.1048
	FLReLU	0.9643	0.0408	0.9702	0.0417
	FPReLU	0.9929	0.0054	0.9932	0.0056
	FELU	0.8075	1.3603	0.8486	1.3801
	FSiLU	0.8342	0.1356	0.9511	0.0431
	FGELU	0.5841	1.1456	0.5981	1.1489
FLReLU	ReLU	0.9346	0.0555	0.9425	0.0563
	LReLU	0.9193	0.0612	0.9277	0.0535
	PReLU	0.9175	0.0703	0.9169	0.0596
	ELU	0.8942	0.1114	0.8743	0.2889
	SiLU	0.9271	0.0546	0.9287	0.0626
	GELU	0.9156	0.0641	0.9131	0.0661
	FReLU	0.9171	0.1065	0.8991	0.1368
	FLReLU	0.9456	0.0409	0.9345	0.0498
	FPReLU	0.8969	0.5441	0.8931	1.4355
	FELU	0.9336	0.0588	0.9226	0.0716
	FSiLU	0.7912	0.1583	0.7844	0.1552
	FGELU	0.525	250.74	0.5618	330
FPReLU	ReLU	0.8794	0.1037	0.8857	0.1098
	LReLU	0.9342	0.0578	0.9323	0.0624
	PReLU	0.9871	0.0215	0.9907	0.0233
	ELU	0.8184	0.1656	0.8169	0.1625
	SiLU	0.9591	0.0908	0.9576	0.0965
	GELU	0.9605	0.0453	0.9617	0.0568
	FReLU	0.8184	0.1326	0.8016	0.1445
	FLReLU	0.9531	0.0401	0.9546	0.0358
	FPReLU	0.8536	0.1235	0.8837	0.1199
	FELU	0.8706	15.522	0.8773	15.632
	FSiLU	0.7428	0.7584	0.7672	0.6945
	FGELU	0.6598	Inf	0.71052	Inf
FELU	ReLU	0.9267	0.1254	0.9317	0.1363
	LReLU	0.8928	0.0784	0.9088	0.0695
	PReLU	0.9407	0.0494	0.9175	0.691
	ELU	0.9475	0.0855	0.9404	0.0901
	SiLU	0.8933	0.1385	0.8849	0.1972
	GELU	0.9054	2.8426	0.9177	3.2352
	FReLU	0.8847	0.1134	0.8918	3.6661
	FLReLU	0.8222	0.1246	0.8391	0.7484
	FPReLU	0.9093	0.0696	0.9006	0.0738
	FELU	0.8558	0.1482	0.8556	0.1392
	FSiLU	0.8485	0.2222	0.8465	2.8651
	FGELU	0.525	250.74	0.5618	262
FSiLU	ReLU	0.8461	0.2991	0.8224	0.311
	LReLU	0.8127	0.46007	0.8017	0.5112
	PReLU	0.1777	121.422	0.1699	80.343
	ELU	0.8296	0.4678	0.839	0.4652
	SiLU	0.8025	0.1408	0.7911	0.1373
	GELU	0.9166	0.06	0.8954	0.0842
	FReLU	0.8603	0.112	0.8536	0.1102
	FLReLU	0.8915	0.0864	0.8736	0.1194
	FPReLU	0.8651	1.0025	0.8461	1.0591
	FELU	0.0789	Inf	0.1243	Inf
	FSiLU	0.203	Inf	0.1989	Inf
	FGELU	0.0588	Inf	0.1122	Inf
FGELU	ReLU	0.4643	13909732.36	0.4621	13909700
	LReLU	0.4033	4299438.79	0.4188	15523900
	PReLU	0.4473	402966327.59	0.4481	402966000
	ELU	0.6321	23082.26	0.6338	23082.3
	SiLU	0.5079	59202784.11	0.4905	59202800
	GELU	0.5591	1203413.61	0.5159	1203410
	FReLU	0.5702	14700123.31	0.5928	14700100
	FLReLU	0.4162	758904.72	0.4531	1620920
	FPReLU	0.5784	11953493.94	0.6239	11953500
	FELU	0.1541	Inf	0.1499	Inf
	FSiLU	0.0829	Inf	0.1043	Inf
	FGELU	0.0904	Inf	0.1392	Inf