Labelling Training Samples Using Crowdsourcing Annotation for Recommendation

<div>Comparisons among KeyRank, IMLK, HILED, and HILI in CRR and CDR. <span class="nowrap"><svg height="10.0062pt" id="M34" style="vertical-align:-0.04980946pt" version="1.1" viewbox="-0.0498162 -9.95639 36.3761 10.0062" width="36.3761pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,4.433,0)"><path d="M861 0V28C774 35 771 41 768 147L759 509C756 612 762 614 851 622V650H681L449 149L221 650H57V622C148 613 153 609 144 479L130 271C123 166 117 123 111 88C104 46 85 34 26 28V0H259V28C192 35 169 42 167 90C166 130 166 173 170 256L185 541H187L411 7H431L675 555H679L683 147C683 41 680 35 598 28V0H861Z"></path></g><g transform="matrix(.013,0,0,-0.013,16.016,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,23.01,0)"><path d="M667 0V28C619 34 599 43 546 100C496 154 386 277 311 372C378 440 449 512 488 547C551 605 575 615 650 622V650H410V622L433 619C478 614 479 604 449 568C404 513 337 443 282 390C254 364 234 350 212 345V520C212 609 222 616 299 622V650H41V622C120 616 128 611 128 520V128C128 41 120 34 44 28V0H304V28C221 34 212 41 212 128V318C229 321 242 318 266 291C355 185 436 87 514 0H667Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,31.878,-5.741)"><path d="M397 380C406 395 404 411 396 425S369 451 350 451C302 451 239 372 192 294H189L199 338C214 407 207 451 180 451C152 451 83 405 30 345L48 318C87 354 117 372 125 372S135 362 127 324L55 -5L61 -12C87 -5 117 3 139 5C154 87 168 162 179 207C198 250 240 310 260 332C281 355 297 366 307 366C321 366 333 360 347 346C351 342 360 342 369 348C379 355 389 366 397 380Z"></path></g></svg>,</span><svg height="10.0062pt" id="M35" style="vertical-align:-0.04980946pt" version="1.1" viewbox="-0.0498162 -9.95639 42.9657 10.0062" width="42.9657pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M729 650H468V622C546 615 554 610 554 527V365H213V527C213 610 221 616 297 622V650H38V622C120 616 128 611 128 527V123C128 39 120 34 40 28V0H303V28C221 34 213 40 213 123V322H554V123C554 39 546 34 460 28V0H728V28C647 34 639 39 639 123V527C639 610 647 615 729 622V650Z"></path></g><g transform="matrix(.013,0,0,-0.013,9.958,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,14.391,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,21.385,0)"><path d="M517 162C503 123 484 88 467 68C445 42 417 34 341 34C291 34 256 34 237 47C219 59 213 81 213 128V317H308C395 317 402 311 415 240H444V431H415C403 364 398 356 307 356H213V584C213 613 215 616 246 616H322C394 616 421 609 435 587C448 566 458 544 467 502L496 506C493 557 488 625 488 650H42V622C120 616 128 612 128 523V125C128 43 120 35 29 28V0H511C520 31 540 125 546 158L517 162Z"></path></g><g transform="matrix(.013,0,0,-0.013,28.769,0)"><path d="M43 650V622C120 616 128 612 128 526V124C128 39 120 33 34 27V0H270C392 0 492 25 567 83C643 141 690 230 690 350C690 444 655 517 605 565C543 625 450 650 323 650H43ZM213 547C213 587 217 598 226 604C236 612 262 617 304 617C371 617 429 604 474 576C554 529 592 439 592 336C592 176 505 36 319 36C246 36 213 55 213 131V547Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,38.468,-5.741)"><path d="M397 380C406 395 404 411 396 425S369 451 350 451C302 451 239 372 192 294H189L199 338C214 407 207 451 180 451C152 451 83 405 30 345L48 318C87 354 117 372 125 372S135 362 127 324L55 -5L61 -12C87 -5 117 3 139 5C154 87 168 162 179 207C198 250 240 310 260 332C281 355 297 366 307 366C321 366 333 360 347 346C351 342 360 342 369 348C379 355 389 366 397 380Z"></path></g></svg>, and <svg height="10.0062pt" id="M36" style="vertical-align:-0.04980946pt" version="1.1" viewbox="-0.0498162 -9.95639 30.4128 10.0062" width="30.4128pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M729 650H468V622C546 615 554 610 554 527V365H213V527C213 610 221 616 297 622V650H38V622C120 616 128 611 128 527V123C128 39 120 34 40 28V0H303V28C221 34 213 40 213 123V322H554V123C554 39 546 34 460 28V0H728V28C647 34 639 39 639 123V527C639 610 647 615 729 622V650Z"></path></g><g transform="matrix(.013,0,0,-0.013,9.958,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,14.391,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,21.385,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,25.915,-5.741)"><path d="M397 380C406 395 404 411 396 425S369 451 350 451C302 451 239 372 192 294H189L199 338C214 407 207 451 180 451C152 451 83 405 30 345L48 318C87 354 117 372 125 372S135 362 127 324L55 -5L61 -12C87 -5 117 3 139 5C154 87 168 162 179 207C198 250 240 310 260 332C281 355 297 366 307 366C321 366 333 360 347 346C351 342 360 342 369 348C379 355 389 366 397 380Z"></path></g></svg> denote the performance of algorithms IMLK, HILED, and HILI in CRR. <span class="nowrap"><svg height="12.308pt" id="M37" style="vertical-align:-0.04979992pt" version="1.1" viewbox="-0.0498162 -12.2582 37.6275 12.308" width="37.6275pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,4.433,0)"><path d="M861 0V28C774 35 771 41 768 147L759 509C756 612 762 614 851 622V650H681L449 149L221 650H57V622C148 613 153 609 144 479L130 271C123 166 117 123 111 88C104 46 85 34 26 28V0H259V28C192 35 169 42 167 90C166 130 166 173 170 256L185 541H187L411 7H431L675 555H679L683 147C683 41 680 35 598 28V0H861Z"></path></g><g transform="matrix(.013,0,0,-0.013,16.016,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,23.01,0)"><path d="M667 0V28C619 34 599 43 546 100C496 154 386 277 311 372C378 440 449 512 488 547C551 605 575 615 650 622V650H410V622L433 619C478 614 479 604 449 568C404 513 337 443 282 390C254 364 234 350 212 345V520C212 609 222 616 299 622V650H41V622C120 616 128 611 128 520V128C128 41 120 34 44 28V0H304V28C221 34 212 41 212 128V318C229 321 242 318 266 291C355 185 436 87 514 0H667Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,31.878,-5.741)"><path d="M538 684C542 703 539 710 529 710C512 710 455 682 363 674L361 644H396C444 644 448 642 438 594L405 438C373 450 347 451 336 451C288 451 200 416 147 374C69 312 24 207 24 113C24 29 60 -12 94 -12C126 -12 156 4 196 30C231 53 296 100 346 166H348L328 81C311 10 325 -12 348 -12C380 -12 450 24 512 98L495 125C466 97 431 70 417 70C408 70 408 80 411 98L538 684ZM390 373L361 240C336 196 210 56 145 56C132 56 113 73 113 131C113 216 156 338 220 379C243 394 266 402 303 402C331 402 375 388 390 373Z"></path></g></svg>,</span> <span class="nowrap"><svg height="12.308pt" id="M38" style="vertical-align:-0.04979992pt" version="1.1" viewbox="-0.0498162 -12.2582 44.2171 12.308" width="44.2171pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M729 650H468V622C546 615 554 610 554 527V365H213V527C213 610 221 616 297 622V650H38V622C120 616 128 611 128 527V123C128 39 120 34 40 28V0H303V28C221 34 213 40 213 123V322H554V123C554 39 546 34 460 28V0H728V28C647 34 639 39 639 123V527C639 610 647 615 729 622V650Z"></path></g><g transform="matrix(.013,0,0,-0.013,9.958,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,14.391,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,21.385,0)"><path d="M517 162C503 123 484 88 467 68C445 42 417 34 341 34C291 34 256 34 237 47C219 59 213 81 213 128V317H308C395 317 402 311 415 240H444V431H415C403 364 398 356 307 356H213V584C213 613 215 616 246 616H322C394 616 421 609 435 587C448 566 458 544 467 502L496 506C493 557 488 625 488 650H42V622C120 616 128 612 128 523V125C128 43 120 35 29 28V0H511C520 31 540 125 546 158L517 162Z"></path></g><g transform="matrix(.013,0,0,-0.013,28.769,0)"><path d="M43 650V622C120 616 128 612 128 526V124C128 39 120 33 34 27V0H270C392 0 492 25 567 83C643 141 690 230 690 350C690 444 655 517 605 565C543 625 450 650 323 650H43ZM213 547C213 587 217 598 226 604C236 612 262 617 304 617C371 617 429 604 474 576C554 529 592 439 592 336C592 176 505 36 319 36C246 36 213 55 213 131V547Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,38.468,-5.741)"><path d="M538 684C542 703 539 710 529 710C512 710 455 682 363 674L361 644H396C444 644 448 642 438 594L405 438C373 450 347 451 336 451C288 451 200 416 147 374C69 312 24 207 24 113C24 29 60 -12 94 -12C126 -12 156 4 196 30C231 53 296 100 346 166H348L328 81C311 10 325 -12 348 -12C380 -12 450 24 512 98L495 125C466 97 431 70 417 70C408 70 408 80 411 98L538 684ZM390 373L361 240C336 196 210 56 145 56C132 56 113 73 113 131C113 216 156 338 220 379C243 394 266 402 303 402C331 402 375 388 390 373Z"></path></g></svg>,</span> and <svg height="12.308pt" id="M39" style="vertical-align:-0.04979992pt" version="1.1" viewbox="-0.0498162 -12.2582 31.6642 12.308" width="31.6642pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M729 650H468V622C546 615 554 610 554 527V365H213V527C213 610 221 616 297 622V650H38V622C120 616 128 611 128 527V123C128 39 120 34 40 28V0H303V28C221 34 213 40 213 123V322H554V123C554 39 546 34 460 28V0H728V28C647 34 639 39 639 123V527C639 610 647 615 729 622V650Z"></path></g><g transform="matrix(.013,0,0,-0.013,9.958,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.013,0,0,-0.013,14.391,0)"><path d="M495 163C480 117 462 85 444 65C421 39 387 34 332 34C290 34 256 36 236 47C218 57 213 77 213 131V526C213 612 222 616 301 622V650H40V622C122 616 128 611 128 526V126C128 41 120 34 36 28V0H489C498 31 519 126 525 157L495 163Z"></path></g><g transform="matrix(.013,0,0,-0.013,21.385,0)"><path d="M303 0V28C221 34 213 39 213 125V525C213 610 221 616 303 622V650H38V622C120 616 128 610 128 525V125C128 40 120 34 38 28V0H303Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,25.915,-5.741)"><path d="M538 684C542 703 539 710 529 710C512 710 455 682 363 674L361 644H396C444 644 448 642 438 594L405 438C373 450 347 451 336 451C288 451 200 416 147 374C69 312 24 207 24 113C24 29 60 -12 94 -12C126 -12 156 4 196 30C231 53 296 100 346 166H348L328 81C311 10 325 -12 348 -12C380 -12 450 24 512 98L495 125C466 97 431 70 417 70C408 70 408 80 411 98L538 684ZM390 373L361 240C336 196 210 56 145 56C132 56 113 73 113 131C113 216 156 338 220 379C243 394 266 402 303 402C331 402 375 388 390 373Z"></path></g></svg> denote the performance of algorithms IMLK, HILED, and HILI in CDR. (a) Group-3. (b) Group-4. (c) Group-5.</div>

Complexity

fig3

Figure 3

Figure 3: Labelling Training Samples Using Crowdsourcing Annotation for Recommendation 