It happened so the very experienced Mr Filipe, from Brazil, came across the post and left me a comment saying that Spoofing over the internet is quiet possible.
I replied surprised, and after a number of comments ping-pongs, we started chatting online, and Felipe had agreed to give me a live spoofing demo:
On my end, I’ve configured my home router to forward TCP/UDP packets to my desktop, where I ran a wireshark network capture to monitor any incoming packets.
Then Felipe sent a burst of packets from random IP source addresses. Proving me that IP spoofing over the Internet is a reality indeed.

(What do you think? Isn’t this kind of stuff is what makes the Internet so amazingly wonderful? two people from two different parts of the world, united by joint interest and kindness )

So, Thank you Filipe!

A few notes on why spoofing might *not* work:

1. According to Filipe, the recipient’s ISP is much more likely to block the spoofed packet, than the sender’s ISP. For example if the recipient’s ISP see a bogon source IP.
   That’s a bit counter-intuitive, because, assuming the ISPs really do care about preventing spoofing, it’s a very easy job for the sender’s ISP to tell if the packet’s source IP is one of the IPs that it handed out to customers, or moreover, to the particular customer (sender).
2. If you are behind a NAT device, then any source address you are planning to use (be it spoofed or real) will be overwritten by the NAT anyway, so make sure you are on a real public IP.
3. No reason to get excited. TCP spoofing is very limited as you won’t make it across the TCP handshake, because the recipients will send their ACK,SYN response to the spoofed IP, which you probably don’t have much control over.
   In a LAN things are a bit different, if you can manipulate the recipient’s ARP table to think that the spoofed IP MAC address is yours. I haven’t dag deep.

Feel free to comment.

The SIP protocol is not NAT oblivious. To traverse NAT our application has to replace the DNS in the SIP message contact header to the external FQDN that the message receiver will be sending responses to (A NAT with static routing configured).
Therefore I needed to test our software in a NAT topology.

In the past, when we used VMWare player/workstation, it had a build-in NAT network. But, unfortunately, the ESX hypervisor does not provide a NATed network option.
Seeking alternatives at VMWare’s appliance marketplace, I found and downloaded the Vyatta’s community edition (VC5) router appliance (also downladble from sourceforge), and comes under the GPL license.
After 3-4 hours – guided by the official quick start guide -  I had a working NAT configuration in the ESX. Hurray!
Overall, not a hard nut to crack , though I wish VMWare will wise up and just add an build-in NAT option to vSphere.

Left to do:
Obtain some static IPs, so the config won’t break each time the vm reboots and the DHCP lease expires.
Tip #1:
If you want want to access your NATed VM by RDP/VNC, without setting up extra NAT routing rules, consider adding the VM an additional un-NATed NIC, but when doing so, make sure that the OS routing tables are set to route through the NIC that is NATed.
Tip #2:
This short vyatta user installation report also helped me a bit.

Here’s the complete configuration script I ended up feeding to the appliance console (network topology is similar to the one presented in the Vyatta’s getting stated guide):
Where:
1.2.3.4 is your department’s DNS server
192.168.1.199 is the VMs NATed private IP address (provided by the DHCP).
The script contains a NAT forward rule for VNC (port 5900)

configure
set system host-name vyatta-nat
set interfaces ethernet eth0 address dhcp
set service ssh
set service https
commit;
save;
# restart the appliance to switch from console remote desktop to SSH:

#login with user and password
configure
show interfaces

set interfaces ethernet eth1 address 192.168.1.254/24

commit;

delete service dhcp-server
set service dhcp-server shared-network-name ETH1_POOL subnet 192.168.1.0/24 start 192.168.1.100 stop 192.168.1.199
set service dhcp-server shared-network-name ETH1_POOL subnet 192.168.1.0/24 default-router 192.168.1.254
set service dhcp-server shared-network-name ETH1_POOL subnet 192.168.1.0/24 dns-server 1.2.3.4
commit;
show service dhcp-server

set service nat rule 1 source address 192.168.1.0/24
set service nat rule 1 outbound-interface eth0
set service nat rule 1 type masquerade
commit;
show service nat
save;
exit
show nat rules
configure
set service nat rule 20 type destination
set service nat rule 20 inbound-interface eth0
# use a negative fake address to so that all incoming communication will be nated
#set service nat rule 20 destination address !192.168.50.0
#Forward traffic to address 192.168.1.199
set service nat rule 20 inside-address address 192.168.1.199
set service nat rule 20 protocol tcp
set service nat rule 20 destination port 5900
commit;
save;
exit

————
UPDATE Sep/2010: Dear Filipe (see comments below) had proven to me that spoofing over the internet is indeed possible, read all about it on the continuation post: My attempts with IP Spoofing – Revisited. Now back to the original story:
————

When customers install our product, they often forget to setup firewall rules to accept incoming connections from public IM (instant messaging) providers. Without the firewall rules in place the product does not function properly, of course, and the customer rushes to open a support trouble ticket. Troubleshooting to pinpoint the problem to a missing firewall rule isn’t trivial. When we try to validate whether the customer defined the required firewall rule, we need the external entity (that we have no control on) to open a connection to the customer’s IP, but the external entity will only do so following the successful completion of a handshake sequence that must be initiated by the customer (consider for example: XMPP Dial-Back mechanism), since this handshake by itself is prone to failures, you can see how reproducing the problem is a combursum process.

I started looking for a simple, independent, and reliable, troubleshooting procedure that would be able to give a clear-cut answer to whether or not the customer defined the firewall correctly.
Here’s what I’ve concocted:

1. Assume that the customer IP is 1.1.1.1 and they were suppose to configure their firewall to allow incoming connections from 2.2.2.2.
2. I’ll send a single TCP SYN packet (the 1^st of the standard three messages TCP handshake) from my computer (say it’s IP is 9.9.9.9), but I’ll spoof the IP datagram’s source address field to be 2.2.2.2 instead of what normally should have been my actual machine address (9.9.9.9).
3. I’ll ask the customer to run a network sniffer on the IM Gateway machine. Waiting for the single packet to arrive at the destination socket.
4. If the sniffer had recorded the incoming IP message, then it means that the firewall is setup correctly and the problem is else where.
   But, If the sniffer didn’t record any incoming SYN packet, then we shell blame the firewall guys.

Pretty simple, eh? Now, in order to spoof the TCP SYN packet I needed a something that could generate and send raw IP packets, since you can’t just fiddle with the source IP address if you choose to ride on the good’ol TCP/IP stack. I found this IP spoofing perl script on the net, and it does the job.

Visualization of the various routes through a portion of the Internet. Took it from Wikipedia.

I did my first test on the office LAN, I sent a message from machine (IP 9.9.9.9) to to machine 1.1.1.1 claiming the message source was 2.2.2.2, it worked! Machine 1.1.1.1 registered an incoming packet from 2.2.2.2.
It seems that the office router went along with the scam, perhaps it thought that the machine switched IP it IP, or the DHCP server went crazy, or that it’s ARP cache is just stall.

In the next test I tried sending the packet over the Internet, I tried sending a packet to my home computer from the office, with a source IP of some foreign entity, to my dismay, it never got to my home computer. Other IP variations didn’t work either.
My guess is that some router along the way noticed that it’s getting a packet with a source IP address that the part of the network it is looking can’t can’t possibly generate (imagine CIDR based ACLs), and that caused it to immediately drop the packet. This failure caused me to give up on the whole spoofing troubleshooting procedure idea.

Some thoughs about what I’ve seen:

1. Evidently, It’s quite trivial to spoofe IP addresses on a LAN.
2. Spoofing  IP addresses over the Internet doesn’t seem to be trivial.
3. A side note: If the customer has a reverese proxy, or any form of entity that delegates TCP handshakes, deployed before the actual IM Gateway machine, then the procedure is not applicable, as the first TCP SYN message will never reach the IM Gateway machine.
4. I would assume that the closer you inject the packet into the Internet backbone blood stream, the better the chances of not getting a rejection of the spoofed packet. The backbone routers communicate with many difference parts of the network, and might not have rational of where certain packates should be coming from or not.
   IP Packets tend to travel in different routes, making it harder to judge what IP CIDR is ligit from each fellow router.
5. I’m guessing that the biggest problem for spoofing is the first or the second router (the ISP’s), since the ISP knows exactly what is your assinged address. Thereby knowning that the packet is spoofed.
6. If any one knows a better method of spoofing source IP, please step forward and share your secret