This was an RSA challenge where you had to break the prime generation function. We see that the function
Generates a 64 bit number
Does a logical OR with 0xc000000000000001 on it
Then, for 16 rounds:
Sets the chunk to s
Multiplies s by a constant a (0xe64a5f84e2762be5)
Then, if the number generated is prime, it returns it
I got s1*s2 by getting the upper and lower bits of n and applying modinv
We can represent p as [x][xa][xa^2]... where each [] is 64 bits long or p = x64^15 + ax64^14...
Multiplying it with q would give (kek how do i represent this) [0 ][1 ][2 ]... ... [30][31] <- chunk indexes (64 bits long) [xy ] ... ... [xya^30] [xya ]... [xya^29] [xya ]... [xya^29] since its n = xy64^30 + 2xya64^30 ... + 2xya^29*64^1 + xya^30
We can get upper and lower bits of n to get upper and lower bits (with modinv of a) of xy like this. however since theres a 2 coefficient on 2xya*64^30 this value is bit shifted and leaks into the last upper bits of xy. this gives 2 values what xy could be (one where the the bit is subtracted and one where it isnt). which are 0xab802dca026b182578adce5060bd0eb1 or 0xab802dca026b182478adce5060bd0eb1
Script below to gen s1*s2 (bit removal hasnt happened yet)
I factored these numbers and generated (kind of) all the 64 bit factors for both of these 128 bit numbers, and then tried using them as s in order to get the primes p and q. 0xab802dca026b182478adce5060bd0eb1 has no 64 bit factors, so it had to be 0xab802dca026b182578adce5060bd0eb1. The 64 bit factors of that number are [17323093358088416319, 11957115919933039605, 15301219884532198649, 14589535238740003363, 10091363333161070837, 14567509746395306455, 15648764542394866359, 15625139955456876315, 14898389259854230905, 14898389259854230905, 15625139955456876315, 15648764542394866359, 14567509746395306455, 10091363333161070837, 14589535238740003363, 15301219884532198649, 11642633479736017985, 9423396846760527157, 9883074741724455311, 13159516333377194255, 12330536802058217123] I then modified the gen_prime function to, if it wasn't prime, don't repeat, since these s's should generate a prime without having to do anything. This resulted in the 2 factors p and q, and then we used this to decrypt the flag.
CTF{__donald_knuths_lcg_would_be_better_well_i_dont_think_s0__}