showmount -e is failing with "clnt_create: RPC: Port mapper failure - Timed out" or "Authentication error" on RHEL 6

Environment

• Red Hat Enterprise Linux 6.4
• NFS

Issue

• A showmount command returns this error . We are able to successfully mount NFS shares from this filer though. This is on a system running RHEL6

$ showmount -e <NFS Server>

clnt_create: RPC: Port mapper failure - Timed out

$ showmount -e <NFS Server>

clnt_create: RPC: Port mapper failure - Authentication error

 Resolution

• Ensure that the NFS server is allowing UDP traffic on port 111.
• Check if rpcbind(portmap) is allowed, otherwise add service in /etc/hosts.allow

rpcbind : client_ip

Root Cause

• The NFS server is not allowing UDP traffic on port 111.
• The showmount program in RHEL 6 uses the RPC functionality provided by the libtirpc library, and libtirpc only uses UDP for portmap GETPORT calls:

$ git show 4c3d6a16b6d1407846192f96d9fd1010c7c3f9a9

commit 4c3d6a16b6d1407846192f96d9fd1010c7c3f9a9

Author: Chuck Lever <chuck.lever@oracle.com>

Date:   Mon Nov 30 08:54:02 2009 -0500

     rpcb_getaddr: Always do PMAP_GETPORT first for NC_INET transports

    For PF_INET transports, use PMAP_GETPORT.  This is what network

    captures show that Solaris user space does, what the Linux mount.nfs

    command does, and what the Linux kernel rpcbind client does.  It's

    more efficient when querying legacy hosts, of which there are still

    many.

    An additional benefit of this change is that since libtirpc uses

    only UDP for PMAP_GETPORT requests, and it now tries PMAP_GETPORT

    first, the creation of a TCP socket in a very common case is now

    avoided.  This reduces the consumption of ephemeral ports that

    can be left in TIME_WAIT after a request.

---8<---

• Prior to RHEL 6, the showmount program used the RPC functionality provided by glibc, which would first try TCP for portmap GETPORT calls.

Diagnostic Steps

• showmount -e using UDP port
• netstat -neopa | grep 111

tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      0          13267      2410/rpcbind        off (0.00/0/0)

·         Tcpdump showed NFS server not responding to UDP traffic for showmount -e.

·         This is what happens when UDP traffic is blocked on port 111 on NFS server:

$ tshark -r rhel6-showmount-port-111-udp-blocked.pcap -R "portmap && tcp"

(nothing)

$ tshark -r rhel6-showmount-port-111-udp-blocked.pcap -R "portmap && udp"

  1   0.000000 192.168.122.130 -> 192.168.122.179 Portmap     V2 GETPORT Call MOUNT(100005) V:3 TCP

  2  15.015531 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #1]V2 GETPORT Call MOUNT(100005) V:3 TCP

  5  30.031113 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #1]V2 GETPORT Call MOUNT(100005) V:3 TCP

  9  45.046738 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #1]V2 GETPORT Call MOUNT(100005) V:3 TCP

 10  60.063210 192.168.122.130 -> 192.168.122.179 Portmap     V2 GETPORT Call MOUNT(100005) V:3 UDP

 11  75.079019 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #10]V2 GETPORT Call MOUNT(100005) V:3 UDP

 14  90.094818 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #10]V2 GETPORT Call MOUNT(100005) V:3 UDP

 18 105.110585 192.168.122.130 -> 192.168.122.179 Portmap     [RPC retransmission of #10]V2 GETPORT Call MOUNT(100005) V:3 UDP

• In a 'working' test machine:-

$ tshark -r rhel6-showmount.pcap -R "portmap && tcp"

(nothing)

$ tshark -r rhel6-showmount.pcap -R "portmap && udp"

  1   0.000000 192.168.122.130 -> 192.168.122.179 Portmap     V2 GETPORT Call MOUNT(100005) V:3 TCP

  2   0.000542 192.168.122.179 -> 192.168.122.130 Portmap     V2 GETPORT Reply (Call In 1) Port:4046

• Verify if NFS server allowing UDP traffic:-

rpcinfo -n 111 -t <address-of-filer> 100000 2

rpcinfo -n 111 -u <address-of-filer> 100000 2