Upgrade Questions on mid 2010 iMac

Upgrading CPU to i7-880s. - Sadly the systems firmware won’t support anything beyond what it was sold with iMac11,2

The heatsink is also not compatible with the i7-880 which is a LGA1156 based chip unlike the i3-540/550 or i5-680 which is a FCLGA1156 based chip.

Upgrading the fans - Apple uses custom fans so there just isn’t anything better.

A better power supply in should handle the heat - Again Apple does their own thing! No options here.

Your system only offers SATA II (3.0 Gb/s) SATA interface. So you are limited to older SATA drives, newer SATA III (6.0 Gb/s) drives can encounter issues if they can’t support the slower 3.0 Gb/s data rate. Review the given drives spec sheet if it doesn’t state it offers SATA II (3.0 Gb/s) compatibility don’t use it! As an example this drive would work in your system Samsung 860 EVO SATA SSD Note the Interface line lists 3.0 Gb/s !!

As you can see looking at the different model specs your system can indeed support 16GB of RAM so that’s something you can do! 1333 MHz PC3-10600 DDR3 SO-DIMMs (204-pin)

Your system came with a Radeon HD 5670 with 512MB of VRAM. You really can’t replace it as you have a few issues that prevent it. First the MXM interface is the older one so you can’t use newer MXM boards (MXM-II) You also need a board that offers Apples firmware to work properly. The systems power and cooling system is not designed to support anything larger. And lastly finding even a Radeon HD 5670 is hard! As they tended to burnout! People would over stress the system running heavy graphical games which the system was not designed to support. They have long since stopped making them, most sold today are not reliable! I’ve stopped putting any MXM GPU board in as finding the correct one even for a 2011 iMac is just too hard and likely to fail soon after I install it (having to rebate the repair fees and loosing the money spend on the part)

You also are limited to an older macOS High Sierra (10.13.x), frankly I would stick with macOS Sierra (10.12) as being the better OS for your system.

OK, I’ve run down every issue you called out, now let’s look at the what if! If you could do everything you wanted to do it would cost more than getting a newer system (new or used) with a everything you want!

So I would focus on the RAM & Storage going with more RAM and a good SSD and calling it a day! Sorry to bust your bubble ;-{