Any pointers on this troublesome algorithm?

[chants]

It looks like it is not encryption/decryption but RSA signature verification/signing:

00000001 13a303bf 04010503 02070609
4207e5e1 00000001 02a00bd1 00000001
ffffff00 ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff ffffffff
ffffffff ffffffff ffffffff 0001ffff

Notice the 0001 which generally indicates PKCS#1 v1.5 RSA signing/private key.

See the RFC for yourself

https://tools.ietf.org/html/rfc2313

on page 9.

What you show is not "decryption" but "verification" using the public exponent of a signature. After all the 0xFF there is a 0x00, then an ASN.1 code and finally a hash of a message being signed.

So you are looking for a private key used to sign messages.

Using chosen plaintext attacks, you might find a way to break this so that you can sign messages without even knowing the private key as long as you can verify signed messages and if the padding is bad. I suggest looking up chosen ciphertext attacks against RSA and look up Bleichenbacher in particular.

If the public exponent is low enough (like e=3), then you can forge messages almost certainly. Using mere small numbers and cube/cube root properties if the padding is not checked properly.