difficulties communicating with the cluster error message after upgrade.

marc,

Please send to me the logs

go to :

http://hostname:8091/index.html#sec=log

Click on Generate Diagnostic Report on the upper right side. Please zip the file that is generated and send it to us.

Also, Can you please run the collect_info command on one of the servers and send to me the output:

/opt/membase/1.6.*/bin/ns_server/collect_info

Can you also please check disk space on all the nodes?

Thanks

Bhawana

Hi Bhawana:

I have attached the log file.

My IT Ops guy found the following in the logs:

Found this in the diagnostic log:<?xml:namespace prefix = o />

memcached<0.105.0>: /opt/membase/1.6.0.1/data/ns_1/isasl.pw: No such file or directory

ERROR REPORT  <0.1914.0>

Put my backed up file into the directory specified, looking better

Our best guess, is there is a step or something else wrong with the upgrade instructions.

marc

This is the point in the process that returned ‘success’ but really doesn’t seem to have done anything:<?xml:namespace prefix = o />

dbupgrade /tmp/membase_data_backup/default /opt/membase/<previous_version>/data/ns_1

after I manually moved the contents of the membase_data_backup dir into the /data/ns_1 dir it started working

Also, all the graphs on the monitor page do not show.  On the big graph I see the timeline moving, but no line is being drawn.  On the various counter boxes, the counts update, but no graph is drawn.

The top keys do refresh.

Please advise.

thanks

marc

marc,

What are the graphs that you are looking at? Did you add any new ones to your view? If you are not actively reading or writing you will not see anything drawn on the default graphs. Please click on 'Day' or 'Week' (on the graph page) so that you can see previous data on the graphs. Do you see the lines on the graphs now?

Bhawana

It doesn't matter what graphs I look at on the monitor page.  None of them draw. 

I see the operations/sec, memory bytes used, items count as well as the list of top keys change, but the graphs just do not draw.

It looks like it is only the minute graphs that do not work.

Marc

I had to delete the bucket I've been using, because I needed to give it more memory.  Now the top keys as well as all the counters are not working.

Please advise.

marc

 Thanks for the update Marc, we're looking into the logs now.

Just to confirm, your cluster has been upgraded and is serving data properly correct?

Thanks

Perry

 Marc, was the last log report before or after you fixed the dirctory issue and got things working?  If it was before, can you run another one now to capture the graphing issues?

Thanks

Perry

Perry, yes I'm able to retrieve items from the cache.  The log report was before the graphing issue.  I'm generating one now.  I will send it via email.

marc

marc,

Can you please send the logs when you get a chance and let us know how is it going.

Thanks

bhawana@membase

Today, I decided to upgrade to the 1.6.5 release.  I uninstalled the 1.6.4.1 release and then cleaned up all the left over directories.

I then installed 1.6.5.  Everything was fine.

This afternoon, I decided to reconfigure the memory assigned to my bucket as I was see a significant number of cache evictions.

I deleted the bucket, recreated it with the new bigger size.

I ultimately ended up rebooting the servers since the client couldn't communicate with the servers.

Now, when I go tot he Data Buckets monitor screen for my bucket, I only see 45 seconds on all the graphs.

I think the graphing getting screwed up has something to do with the bucket being deleted and recreated.

Please let me know where to send the diag report.

thanks

marc

 Thanks Marc, send it over to perry -at- membase -dot- com and we'll take a look.

Perry

Perry, I just wanted to let yo know that setting up NTP clients on the servers, seemed to resolve the graphing problems.

marc

 Thanks for the feedback Marc, glad we were able to get it worked out for you.  Incidentally, this is documented in our release notes as well: http://wiki.membase.org/display/membase/Membase+Server+1.6.5.  Don't worry about it, I never read those things anyway :P