how to remove square shadows behind the dock icons in i os4.0

The files are located in System/Library/CoreServices/Springboard.app:

WallpaperIconDockShadow.png
and
WallpaperIconDockShadowT.png

Not sure which one you need modified, but you can replace them with a blank png.
Apparently, IconDarkeningOverlay.png is used for the icon overlay when icon is pressed.