Limiting Dynamics for Q-Learning with Memory One in Symmetric Two-Player, Two-Action Games

<div>Diagram showing the structure of an interaction between the two players during a batch. The batch size is <svg height="8.68572pt" id="M363" style="vertical-align:-0.0498209pt" version="1.1" viewbox="-0.0498162 -8.6359 9.95144 8.68572" width="9.95144pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M743 650H503L496 622L527 618C563 613 564 603 532 573C449 495 371 431 323 392C301 374 272 355 246 346L280 522C297 609 300 614 379 622L385 650H135L129 622C209 614 215 609 198 522L124 133C106 39 99 35 23 28L17 0H271L277 28C193 35 192 39 208 133L239 316C264 328 280 325 303 288C368 183 435 90 502 0H652L659 28C602 34 584 43 543 94C495 154 403 283 347 369L574 554C634 603 659 612 735 624L743 650Z"></path></g></svg> so that <svg height="11.5564pt" id="M364" style="vertical-align:-2.26807pt" version="1.1" viewbox="-0.0498162 -9.28833 71.7587 11.5564" width="71.7587pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M480 416C480 431 465 448 438 448C388 448 312 383 252 330C217 299 188 273 155 237H153L257 680C262 700 263 712 253 712C240 712 183 684 97 674L92 648L126 647C166 646 172 645 163 606L23 -6L29 -12C51 -5 77 2 107 8C115 62 130 128 142 180C153 193 179 220 204 241C231 170 259 106 288 54C317 0 336 -12 358 -12C381 -12 423 2 477 80L460 100C434 74 408 54 398 54C385 54 374 65 351 107C326 154 282 241 263 299C296 332 351 377 403 377C424 377 436 372 445 368C449 366 456 368 462 375C472 386 480 402 480 416Z"></path></g><g transform="matrix(.013,0,0,-0.013,10.171,0)"><path d="M448 1V51H364C248 51 153 129 140 230H448V280H140C153 381 248 459 364 459H448V509H365C208 509 80 395 80 255S208 1 365 1H448Z"></path></g><g transform="matrix(.013,0,0,-0.013,20.667,0)"><path d="M293 -169V-141C218 -131 209 -90 209 -44C209 19 222 85 222 152C222 207 198 255 139 269V273C199 286 222 334 222 388C222 454 209 523 209 588C209 632 218 671 293 681V709C234 709 148 695 148 577C147 513 155 438 155 372C155 337 152 291 64 285V256C152 250 155 204 155 169C156 105 148 31 148 -41C148 -157 234 -169 293 -169Z"></path></g><g transform="matrix(.013,0,0,-0.013,25.178,0)"><path d="M384 0V27C293 34 287 42 287 114V635C232 613 172 594 109 583V559L157 557C201 555 205 550 205 499V114C205 42 199 34 109 27V0H384Z"></path></g><g transform="matrix(.013,0,0,-0.013,31.418,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z"></path></g><g transform="matrix(.013,0,0,-0.013,36.561,0)"><path d="M113 -12C146 -12 170 11 170 46C170 78 146 103 114 103S58 78 58 46C58 11 82 -12 113 -12Z"></path></g><g transform="matrix(.013,0,0,-0.013,41.704,0)"><path d="M113 -12C146 -12 170 11 170 46C170 78 146 103 114 103S58 78 58 46C58 11 82 -12 113 -12Z"></path></g><g transform="matrix(.013,0,0,-0.013,46.847,0)"><path d="M113 -12C146 -12 170 11 170 46C170 78 146 103 114 103S58 78 58 46C58 11 82 -12 113 -12Z"></path></g><g transform="matrix(.013,0,0,-0.013,52.024,0)"><path d="M95 130C70 130 46 113 46 88C46 72 54 64 59 64C93 55 121 33 121 -3C121 -41 93 -68 44 -88L55 -117C117 -98 186 -56 186 22C186 91 131 130 95 130Z"></path></g><g transform="matrix(.013,0,0,-0.013,57.167,0)"><path d="M743 650H503L496 622L527 618C563 613 564 603 532 573C449 495 371 431 323 392C301 374 272 355 246 346L280 522C297 609 300 614 379 622L385 650H135L129 622C209 614 215 609 198 522L124 133C106 39 99 35 23 28L17 0H271L277 28C193 35 192 39 208 133L239 316C264 328 280 325 303 288C368 183 435 90 502 0H652L659 28C602 34 584 43 543 94C495 154 403 283 347 369L574 554C634 603 659 612 735 624L743 650Z"></path></g><g transform="matrix(.013,0,0,-0.013,66.984,0)"><path d="M283 255V284C195 290 192 337 192 375C192 436 200 508 200 580C200 696 116 709 54 709V681C127 671 139 634 139 588C138 524 125 452 125 387C125 333 149 286 209 272V268C148 253 125 206 125 151C126 87 139 16 139 -47C139 -92 127 -131 54 -141V-169C115 -169 200 -152 200 -41C200 32 192 104 192 164C192 202 195 249 283 255Z"></path></g></svg> and <svg height="15.0098pt" id="M365" style="vertical-align:-3.9436pt" version="1.1" viewbox="-0.0498162 -11.0662 20.7424 15.0098" width="20.7424pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M699 368C699 549 574 666 407 666C186 666 23 488 23 277C23 113 129 -3 288 -13L307 -26C431 -111 501 -139 533 -147C559 -154 613 -163 658 -164L666 -141C597 -111 507 -66 430 -11L416 -1C580 42 699 190 699 368ZM601 371C601 227 518 54 381 22L354 40L278 24C175 47 120 145 120 269C120 451 235 631 398 631C540 631 601 521 601 371Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,9.386,-5.741)"><path d="M433 39L423 65C413 59 399 54 387 54C370 54 352 69 352 114V299C352 352 342 392 307 422C285 440 255 449 225 449C168 437 102 399 75 379C56 365 44 353 44 339C44 315 69 296 87 296C101 296 111 303 116 319C124 349 133 371 145 385C156 397 171 404 190 404C241 404 275 364 275 291V274C253 256 180 229 120 209C65 190 39 159 39 110C39 47 88 -12 159 -12C189 -12 237 25 277 52C282 35 288 21 301 8C312 -3 333 -12 348 -12L433 39ZM275 84C256 65 221 48 195 48C164 48 124 73 124 124C124 161 146 180 185 198C206 208 254 229 275 240V84Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,13.381,-5.741)"><path d="M390 111C344 68 312 56 269 56C212 56 118 102 118 241C118 346 175 401 241 401C277 401 312 388 342 360C350 352 355 349 361 349C372 349 394 371 394 392C394 403 391 411 378 422C362 436 329 449 288 449H287C250 449 190 432 138 392C71 341 37 274 37 197C37 90 112 -12 238 -12C297 -12 363 32 407 90L390 111Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,17.294,-5.741)"><path d="M298 36L289 62C276 55 253 45 228 45C202 45 169 60 169 141V397H276C289 405 292 426 282 437H169V574L155 576L90 509V437H45L17 408L21 397H90V107C90 28 125 -12 188 -12C198 -12 213 -8 230 1L298 36Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,9.321,3.784)"><path d="M490 97L476 124C442 96 405 70 398 70C392 70 390 78 396 114C419 243 448 379 463 432L457 436C446 436 431 439 418 442C393 447 368 451 343 451C281 451 204 418 155 381C74 320 24 206 24 107C24 23 59 -12 88 -12C118 -12 155 5 191 34C236 70 290 122 328 177H330L312 84C296 0 311 -12 331 -12C355 -12 425 24 490 97ZM374 387C371 367 360 299 347 264C323 202 187 53 142 53C128 53 113 73 113 120C113 224 157 332 221 380C241 395 274 403 303 403C330 403 360 395 374 387Z"></path></g></svg> remains fixed during the batch, while <svg height="16.2154pt" id="M366" style="vertical-align:-3.943501pt" version="1.1" viewbox="-0.0498162 -12.2719 20.6145 16.2154" width="20.6145pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><g transform="matrix(.013,0,0,-0.013,0,0)"><path d="M699 368C699 549 574 666 407 666C186 666 23 488 23 277C23 113 129 -3 288 -13L307 -26C431 -111 501 -139 533 -147C559 -154 613 -163 658 -164L666 -141C597 -111 507 -66 430 -11L416 -1C580 42 699 190 699 368ZM601 371C601 227 518 54 381 22L354 40L278 24C175 47 120 145 120 269C120 451 235 631 398 631C540 631 601 521 601 371Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,9.386,-5.741)"><path d="M478 437H300V411C357 404 363 393 347 345C325 280 293 190 258 107C229 178 188 284 163 354C147 397 149 405 203 411V437H-2V411C52 403 59 396 82 339C130 222 176 109 220 -11H248C304 138 356 261 390 336C417 395 425 404 478 411V437Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,13.6,-5.741)"><path d="M433 39L423 65C413 59 399 54 387 54C370 54 352 69 352 114V299C352 352 342 392 307 422C285 440 255 449 225 449C168 437 102 399 75 379C56 365 44 353 44 339C44 315 69 296 87 296C101 296 111 303 116 319C124 349 133 371 145 385C156 397 171 404 190 404C241 404 275 364 275 291V274C253 256 180 229 120 209C65 190 39 159 39 110C39 47 88 -12 159 -12C189 -12 237 25 277 52C282 35 288 21 301 8C312 -3 333 -12 348 -12L433 39ZM275 84C256 65 221 48 195 48C164 48 124 73 124 124C124 161 146 180 185 198C206 208 254 229 275 240V84Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,17.64,-5.741)"><path d="M238 0V26C174 32 166 38 166 104V712C132 700 70 683 18 677V653C81 647 87 645 87 577V104C87 38 78 32 15 26V0H238Z"></path></g><g transform="matrix(.0091,0,0,-0.0091,9.321,3.784)"><path d="M490 97L476 124C442 96 405 70 398 70C392 70 390 78 396 114C419 243 448 379 463 432L457 436C446 436 431 439 418 442C393 447 368 451 343 451C281 451 204 418 155 381C74 320 24 206 24 107C24 23 59 -12 88 -12C118 -12 155 5 191 34C236 70 290 122 328 177H330L312 84C296 0 311 -12 331 -12C355 -12 425 24 490 97ZM374 387C371 367 360 299 347 264C323 202 187 53 142 53C128 53 113 73 113 120C113 224 157 332 221 380C241 395 274 403 303 403C330 403 360 395 374 387Z"></path></g></svg> is updated.</div>

Complexity

fig9

Figure 9

Figure 9: Limiting Dynamics for Q-Learning with Memory One in Symmetric Two-Player, Two-Action Games 