| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-14 | |||
| 20:29:07 | dims | mtreinish : sure http://logs.openstack.org/57/493057/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/3004b34/logs/grenade.sh.txt.gz#_2017-08-14_17_21_46_226 | |
| 20:29:41 | dansmith | smcginnis: because either n-cpu is not running, or it's not on the right mq bus | |
| 20:30:08 | smcginnis | dansmith: So might still just be a devstack setup issue? | |
| 20:30:29 | dansmith | smcginnis: all you're changing is setup, so it's clearly a setup issue :) | |
| 20:30:47 | dims | dansmith : mtreinish : you can see "Begin detaching volume completed successfully." in the new/screen-c-api.txt grep for "req-2e9c3e74-be5f-4d52-8ac9-878da1b6836e" | |
| 20:30:55 | dansmith | right | |
| 20:31:07 | smcginnis | dansmith: Fair point. ;) | |
| 20:31:20 | dansmith | the next thing after begin is an rpc cast down to the compute | |
| 20:31:25 | dansmith | which never seems to make it | |
| 20:31:55 | dims | aha dansmith | |
| 20:33:03 | smcginnis | dansmith: Which leads to what you were saying of n-cpu being down or on the wrong bus. | |
| 20:33:13 | dansmith | yes | |
| 20:33:35 | dansmith | and that we see no evidence of n-cpu activity around the time that is happening, in any of the scattered logs | |
| 20:33:46 | dims | dansmith : which nova process picks it up? | |
| 20:34:02 | mtreinish | dims: I'm more concerned with it doesn't seem to be starting the services up again. All I see is systemd logging the service stop, not the start up again | |
| 20:34:13 | dims | mtreinish : look under new | |
| 20:34:20 | dims | that's where the new logs are | |
| 20:34:21 | dansmith | also | |
| 20:34:26 | dansmith | n-cpu's config has the wrong rabbit vhost: http://logs.openstack.org/57/493057/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/3004b34/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 20:34:29 | dansmith | see transport_url in there | |
| 20:34:59 | dansmith | dims: we're exporting that CELLSv2_SETUP variable in the existing upgrade path.. maybe you need to do that instead of setting it in devstack_localrc | |
| 20:35:00 | dansmith | ? | |
| 20:35:47 | clarkb | mtreinish: dims itdefinitely does look like grenade may need to update how it configures logging so that things go to stdout and get journald'd rather than ending up in the log file | |
| 20:36:05 | smcginnis | That would be nice. | |
| 20:36:13 | dims | dansmith : add a from-pike/upgrade-nova? | |
| 20:36:20 | mtreinish | clarkb: yeah the logging is all messed up | |
| 20:36:31 | dansmith | dims: idk, maybe? | |
| 20:37:19 | dims | ok, any other suggestions? | |
| 20:37:45 | dansmith | my other suggestion was exporting it instead of setting it in localrc, | |
| 20:38:22 | dansmith | thusly: https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/from-ocata/upgrade-nova | |
| 20:38:36 | dansmith | I think from-pike will run too late to help you here, but I could be wrong | |
| 20:38:43 | dansmith | you know | |
| 20:38:44 | dansmith | actually | |
| 20:38:47 | dansmith | maybe that is what we need | |
| 20:38:55 | dims | ack, will try that | |
| 20:39:17 | clarkb | mtreinish: grenade runs ensure_logs_exist which is something we'll need to probably just remove | |
| 20:41:22 | dims | dansmith : one sec.... that will run base devstack (stable/pike) with superconductor and then switch the master devstack to singleconductor | |
| 20:41:31 | dims | is that what you want to try? | |
| 20:42:03 | dims | not sure if that will work dansmith | |
| 20:44:14 | mtreinish | clarkb: yeah probably, that was added as a sanity check but it's probably not needed now | |
| 20:44:41 | clarkb | also looking again the two systemctl starts for n-cpu may be primary node and subnode | |
| 20:44:59 | clarkb | doesn't look like we upgrade n-cpu until 17:18ish and not seeing systemctl start there so possibly we are not starting it with systemd | |
| 20:45:15 | mtreinish | clarkb: unless there is a way to check it with journalctl sanely (which would be tricky because the journal will show the logs from the old side) | |
| 20:45:42 | mtreinish | hmm, yeah that's what I was thinking based on looking at the logs before | |
| 20:45:50 | clarkb | mtreinish: probably want to check there are log contents in journalctl from after it last restarted | |
| 20:46:17 | mtreinish | clarkb: heh, right which probably requires tracking the time stamps or something | |
| 20:46:48 | dims | smcginnis : the transport_url is definitely different between http://logs.openstack.org/57/493057/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/3004b34/logs/etc/cinder/cinder.conf.txt.gz and http://logs.openstack.org/57/493057/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/3004b34/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 20:47:20 | dims | transport_url = rabbit://stackrabbit:secretrabbit@15.184.65.177:5672/nova_cell1 and transport_url = rabbit://stackrabbit:secretrabbit@15.184.65.177:5672/ | |
| 20:47:38 | clarkb | mtreinish: systemd will tell you | |
| 20:47:41 | clarkb | I think | |
| 20:48:19 | dims | smcginnis : dansmith : i think i'll go with trying to patch that up first (rather than the CELLSv2_SETUP superconductor -> singleconductor) | |
| 20:49:10 | mtreinish | clarkb: that's what I wasn't sure of | |
| 20:54:38 | smcginnis | dims: Looks like a good plan. | |
| 20:54:48 | clarkb | mtreinish: looks like a source of a localrc in 50_neutron/upgrade.sh unsets USE_SYSTEMD | |
| 20:56:58 | clarkb | mtreinish: I see it, patch in a sec | |
| 20:58:35 | dims | smcginnis : updated 493380 (WIP) and rechecked 493057 | |
| 21:00:10 | smcginnis | dims: Great, fingers crossed. | |
| 21:00:20 | clarkb | mtreinish: https://review.openstack.org/493673 that will likely need to depends on a grenade change that removes ensure_logs_exist then after those two go in add back in journald checking | |
| 21:01:27 | mtreinish | clarkb: just approved it | |
| 21:01:45 | mtreinish | we can rebase dim's grenade branch, or just wait | |
| 21:01:47 | clarkb | mtreinish: well it will fail grenade I think due to ensure_logs_exist | |
| 21:01:57 | smcginnis | dims: Hmm, doesn't seem right that cinder.conf transport_url would need to be changed to a nova_cell. | |
| 21:02:28 | mtreinish | clarkb: hmm, that's a good point. Let's see how it turns out | |
| 21:06:21 | dims | mtreinish : please don't rebase :) | |
| 21:10:57 | mtreinish | clarkb: https://review.openstack.org/#/c/493677/ should remove the ensure logs calls | |
| 21:11:36 | clarkb | mtreinish: do you want me to go ahead nad depend on that now or wait for mine to fail on those checks to confirm they will cause problems first? | |
| 21:12:26 | mtreinish | clarkb: probably better to wait and update, just in case we don't actually need it | |
| 21:12:51 | clarkb | ok | |
| 21:17:07 | openstackgerrit | Michael Still proposed openstack/nova master: WIP: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 21:19:53 | openstackgerrit | Michael Still proposed openstack/nova master: WIP: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 21:24:13 | openstackgerrit | Merged openstack/nova master: doc: Extend nfv feature matrix with pinning/NUMA https://review.openstack.org/327126 | |
| 22:06:37 | clarkb | mtreinish: http://logs.openstack.org/73/493673/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/7e5d3b8/logs/grenade.sh.txt.gz#_2017-08-14_21_57_18_157 it failed as expected, adding the depends-on now | |
| 22:07:24 | mtreinish | clarkb: ok cool, I'll fast approve both after your respin | |
| 22:07:57 | mtreinish | unless dansmith wants to review the grenade side patch | |
| 22:12:07 | dansmith | mtreinish: review which? | |
| 22:12:59 | mtreinish | dansmith: https://review.openstack.org/493677 | |
| 22:13:54 | dansmith | mtreinish: ah okay.. you want me to +2+W that? | |
| 22:14:01 | mtreinish | dansmith: sure | |
| 22:14:02 | dansmith | I trust your logic | |
| 22:16:13 | mtreinish | dansmith: modifiy isn't a real word? :) | |
| 22:17:01 | dansmith | mtreinish: modif-i-y like d-i-y maybe.. you could make up a definition | |
| 22:23:49 | clarkb | mtreinish: dansmith hrm http://logs.openstack.org/73/493673/1/check/gate-grenade-dsvm-neutron-ubuntu-xenial/7e5d3b8/logs/old/screen-q-l3.txt.gz is not how I expected it, I expected that to go in journald | |
| 22:24:07 | clarkb | so maybe that devstack change isn't quite what we need either, but in any case we definitely need to not error on missing log files while we sort this out | |
| 22:25:34 | mtreinish | clarkb: could it be d-g doing the collection? | |
| 22:25:37 | clarkb | heh that is beacuse base branch was ocata in that change | |
| 22:25:42 | clarkb | not pike | |
| 22:25:45 | mtreinish | ah, ok | |
| 22:26:00 | clarkb | so there may be an ordering here was need to accomodate, but I think change is otherwise sound | |
| 22:26:21 | clarkb | how did that other job end up being pike to master? | |
| 22:28:00 | clarkb | mtreinish: https://review.openstack.org/#/c/493057/11/devstack-vm-gate-wrap.sh I think we want to depends on that change as well? | |
| 22:28:15 | clarkb | and just live with the weirdness until ^ is in | |
| 22:31:02 | smcginnis | clarkb: Yep, that patch is what sets up the pike to master. ^ | |
| 22:31:20 | clarkb | ok let me push a depends on that too and we'll just live with it a bit | |
| 22:31:42 | clarkb | oh wait we already did it the othe rway around | |
| 22:31:46 | clarkb | in that case onward :) | |
| 22:32:04 | smcginnis | clarkb: Or it can be rebased. Whatever makes it easier to figure out the right way here. | |
| 22:32:26 | clarkb | no thats fine they are in different repos so depends-on is what works | |
| 22:32:35 | openstackgerrit | Michael Still proposed openstack/nova master: WIP: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 22:53:29 | clarkb | mtreinish: smcginnis also is this going to be a place where we need rolling changes to grenade? | |
| 22:53:42 | clarkb | I guess not becuse we only check logs on the target size | |