Couchbase Server

ns_server is still validating ip address in ip file even if erlang already has node name defined (was: 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 2.0-beta-2, 2.0
  • Fix Version/s: 2.0
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None

Description


Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

Error messages from the log:
[ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
[ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

The logs are available in the link below:
https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip

update:

Apparently as part of process of setting up node name folks just left original /opt/couchbase/var/lib/couchbase/ip. And ns_server's bug is due to attempt to validate that address even though it won't be actually used.

Activity

Ascending order - Click to sort in descending order
Bala Keelapudi made changes -
Field Original Value New Value
Assignee Abhinav Dangeti [ abhinav ]
Ketaki Gangal made changes -
Fix Version/s 2.0 [ 10114 ]
Affects Version/s 2.0 [ 10114 ]
Abhinav Dangeti made changes -
Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
Hide
Permalink
Steve Yen added a comment -
abhinav now attempting to reproduce
Show
Steve Yen added a comment - abhinav now attempting to reproduce
Steve Yen made changes -
Summary 2.0 Bulid 1941 : Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.
Assignee Aleksey Kondratenko [ alkondratenko ] Abhinav Dangeti [ abhinav ]
Priority Major [ 3 ] Blocker [ 1 ]
Hide
Permalink
Steve Yen added a comment -
please also get DNS diagnostic info... like ping (ask alk).

it could be as simple as DNS propagation delay.
Show
Steve Yen added a comment - please also get DNS diagnostic info... like ping (ask alk). it could be as simple as DNS propagation delay.
Hide
Permalink
Aleksey Kondratenko added a comment -
From error message it appears that DNS resolver still thinks old ip is assigned to this hostname.

So in order to help diagnosing this I need both cbcollect_info (or just output of ifconfig -a) and some information about in what ip this hostname is resolved. Simple way is by pinging hostname and sending me output
Show
Aleksey Kondratenko added a comment - From error message it appears that DNS resolver still thinks old ip is assigned to this hostname. So in order to help diagnosing this I need both cbcollect_info (or just output of ifconfig -a) and some information about in what ip this hostname is resolved. Simple way is by pinging hostname and sending me output
Hide
Permalink
Abhinav Dangeti added a comment -
- So started with 10.1.3.235, 10.1.3.236 (build 1954)
- set host ip's on /etc/hosts
- stopped couchbase-server on 10.1.3.236
- changed ip of 10.1.3.236 to 10.1.3.222
- resolved /etc/hosts to the new ip
- started couchbase-server back up on 10.1.3.222, server never comes back up.

[ Servers available as is ]

10.1.3.222>>
ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:97:02:D2
          inet addr:10.1.3.222 Bcast:10.255.255.255 Mask:255.0.0.0
          inet6 addr: fe80::250:56ff:fe97:2d2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:109556236 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108558164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:83579290130 (77.8 GiB) TX bytes:86321632329 (80.3 GiB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:72816990 errors:0 dropped:0 overruns:0 frame:0
          TX packets:72816990 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:120374350544 (112.1 GiB) TX bytes:120374350544 (112.1 GiB)

sit0 Link encap:IPv6-in-IPv4
          NOARP MTU:1480 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

<<Attaching cbcollectinfo_10_1_3_222.zip>>
Show
Abhinav Dangeti added a comment - - So started with 10.1.3.235, 10.1.3.236 (build 1954) - set host ip's on /etc/hosts - stopped couchbase-server on 10.1.3.236 - changed ip of 10.1.3.236 to 10.1.3.222 - resolved /etc/hosts to the new ip - started couchbase-server back up on 10.1.3.222, server never comes back up. [ Servers available as is ] 10.1.3.222>> ifconfig -a eth0 Link encap:Ethernet HWaddr 00:50:56:97:02:D2           inet addr:10.1.3.222 Bcast:10.255.255.255 Mask:255.0.0.0           inet6 addr: fe80::250:56ff:fe97:2d2/64 Scope:Link           UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1           RX packets:109556236 errors:0 dropped:0 overruns:0 frame:0           TX packets:108558164 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:83579290130 (77.8 GiB) TX bytes:86321632329 (80.3 GiB) lo Link encap:Local Loopback           inet addr:127.0.0.1 Mask:255.0.0.0           inet6 addr: ::1/128 Scope:Host           UP LOOPBACK RUNNING MTU:16436 Metric:1           RX packets:72816990 errors:0 dropped:0 overruns:0 frame:0           TX packets:72816990 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:0           RX bytes:120374350544 (112.1 GiB) TX bytes:120374350544 (112.1 GiB) sit0 Link encap:IPv6-in-IPv4           NOARP MTU:1480 Metric:1           RX packets:0 errors:0 dropped:0 overruns:0 frame:0           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:0           RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) <<Attaching cbcollectinfo_10_1_3_222.zip>>
Abhinav Dangeti made changes -
Attachment cbcollectinfo_10_1_3_222.zip [ 15859 ]
Abhinav Dangeti made changes -
Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
Hide
Permalink
Aleksey Kondratenko added a comment -
a) cannot access .222.

b) don't have ping output that I need in order to understand more
Show
Aleksey Kondratenko added a comment - a) cannot access .222. b) don't have ping output that I need in order to understand more
Aleksey Kondratenko made changes -
Assignee Aleksey Kondratenko [ alkondratenko ] Abhinav Dangeti [ abhinav ]
James Mauss made changes -
Description
Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

Error messages from the log:
[ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
[ns_server:warn,2012-11-16T13:16:45.522,ns_1@couch14.dev.jawfishgames.com:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

The logs are available in the link below:
https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip







Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

Error messages from the log:
[ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
[ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

The logs are available in the link below:
https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip






Hide
Permalink
Abhinav Dangeti added a comment - - edited
If we use ifconfig eth0 10.1.3.222, to change the IP, we see the issue.
However the issue doesn't occur when the IP is changed this way:
vim /etc/sysconfig/network-scripts/ifcfg-eth0

(and comment out the BOOTPROTO=dhcp and set it to static)
..
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
#BOOTPROTO=dhcp
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.1.3.222
GATEWAY=10.1.0.1
NETMASK=255.255.0.0
...
sudo /etc/init.d/network restart

The reason why this worked was because /opt/couchbase/var/lib/couchbase/ip was empty.
Show
Abhinav Dangeti added a comment - - edited If we use ifconfig eth0 10.1.3.222, to change the IP, we see the issue. However the issue doesn't occur when the IP is changed this way: vim /etc/sysconfig/network-scripts/ifcfg-eth0 (and comment out the BOOTPROTO=dhcp and set it to static) .. # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) DEVICE=eth0 #BOOTPROTO=dhcp BOOTPROTO=static ONBOOT=yes IPADDR=10.1.3.222 GATEWAY=10.1.0.1 NETMASK=255.255.0.0 ... sudo /etc/init.d/network restart The reason why this worked was because /opt/couchbase/var/lib/couchbase/ip was empty.
Hide
Permalink
Aleksey Kondratenko added a comment -
Looks like /opt/couchbase/var/lib/.../ip is still being used somehow. I recommend manually deleting it. It's still ns_server's bug if we try do anything about it when hostname is specificied
Show
Aleksey Kondratenko added a comment - Looks like /opt/couchbase/var/lib/.../ip is still being used somehow. I recommend manually deleting it. It's still ns_server's bug if we try do anything about it when hostname is specificied
Aleksey Kondratenko made changes -
Summary 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. ns_server is still validating ip address in ip file even if erlang already has node name defined (was: 2.0 Build 1941: Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one.)
Description
Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

Error messages from the log:
[ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
[ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

The logs are available in the link below:
https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip







Couchbase Server does not start after a change in IP, server is looking for the old IP even after the hostname resolves to the new one. Followed the best-practices information to configure the hostname in the couchbase-server file and this issue is reproducible in the 1941 2.0 build.

Error messages from the log:
[ns_server:info,2012-11-16T13:16:45.502,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:read_address_config:55]Reading ip config from "/opt/couchbase/var/lib/couchbase/ip"
[ns_server:warn,2012-11-16T13:16:45.522,ns_1@FQDN:dist_manager<0.2732.0>:dist_manager:is_good_address:81]Cannot listen on address `OLD IP`: eaddrnotavail

The logs are available in the link below:
https://s3.amazonaws.com/customers.couchbase.com/jawfishgames/couch14-build-1914.zip

update:

Apparently as part of process of setting up node name folks just left original /opt/couchbase/var/lib/couchbase/ip. And ns_server's bug is due to attempt to validate that address even though it won't be actually used.
Aleksey Kondratenko made changes -
Assignee Abhinav Dangeti [ abhinav ] Aleksey Kondratenko [ alkondratenko ]
Steve Yen made changes -
Component/s ns_server [ 10019 ]
Hide
Permalink
Steve Yen added a comment -
The ip file was/is being used by cbupgrade situation. Please see...

  http://www.couchbase.com/issues/browse/MB-7241
Show
Steve Yen added a comment - The ip file was/is being used by cbupgrade situation. Please see...    http://www.couchbase.com/issues/browse/MB-7241
Hide
Permalink
Aleksey Kondratenko added a comment -
Approved for 2.0. Be careful with using right branch
Show
Aleksey Kondratenko added a comment - Approved for 2.0. Be careful with using right branch
Aleksey Kondratenko made changes -
Assignee Aleksey Kondratenko [ alkondratenko ] Aliaksey Artamonau [ Aliaksey Artamonau ]
Hide
Permalink
Steve Yen added a comment - - edited
i think this fix was merged? -- http://review.couchbase.org/#/c/22895/
Show
Steve Yen added a comment - - edited i think this fix was merged? -- http://review.couchbase.org/#/c/22895/
Aliaksey Artamonau made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]

People

  • Assignee:
    Aliaksey Artamonau
    Reporter:
    Bala Keelapudi
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: