Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
20:10:20 cdent but I nipped it
20:10:30 bauzas anyway, time will see
20:11:07 bauzas the problem is not about having a mutable conf opt :)
20:11:34 mriedem cdent: good - that's how i first read that one, until i re-read it
20:11:35 bauzas it's rather about making sure nova can sighub
20:11:45 kashyap dansmith: So, on that potential race-thing, did you see the comment with more Neutron-educated Sean Mooney? He makes a good point when he says:
20:11:54 kashyap "it would be nicer to have a solution that is always guaranteed to work but even with a race it much less likely to be noticed then the race we have today were the RARP packets from qemu are often lost as the nic is not wired up."
20:11:56 mriedem sighub - the special interest group for github
20:12:02 bauzas oops
20:12:09 bauzas sighup :p
20:12:15 bauzas kill -1 :p
20:13:34 bauzas mriedem: oh btw. why have I a plumage for our team ? :p
20:14:09 mriedem bauzas: you're the only one with a high enough plumage score
20:14:20 mriedem french
20:14:28 mriedem mandatory minimum level of plumage
20:15:20 dansmith kashyap: that has nothing to do with the race I was asking about
20:15:37 bauzas mriedem: because we are all cocks ? :)
20:15:49 mriedem ha
20:16:03 kashyap dansmith: Okay, I see -- you're asking the impact of setting it in two places
20:16:15 dansmith kashyap: yes and in two threads
20:16:22 odyssey4me mriedem actually, we have some logs from just the standard upgrade test: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/
20:16:23 kashyap So in the existing code in _live_migration(), guest.migrate() is already called with 'CONF.libvirt.live_migration_bandwidth'. And then...
20:16:41 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Isolate placement database config https://review.openstack.org/541435
20:16:49 kashyap dansmith: A test result with log would convince you I suppose?
20:16:50 odyssey4me mriedem the 'host' folder will have the nova-compute logs, and the 'openstack' folder will have the other nova services logs
20:17:02 dansmith kashyap: no of course not.. if it's a race then a passing test means nothing :)
20:17:03 mriedem odyssey4me: holy wingdings batman http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-api-os-compute.log.txt.gz.txt.gz
20:17:15 odyssey4me ouch
20:17:17 mriedem oh here we go http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-api-os-compute.log.txt.gz
20:17:19 kashyap dansmith: An automated test that repeats, say 100 times?
20:17:29 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: TrivialFix: Add a space between messages https://review.openstack.org/541436
20:17:30 dansmith kashyap: are you joking? I can't tell
20:17:33 kashyap :P
20:17:40 kashyap Okay, it's clearly too late for me
20:17:51 odyssey4me looks like we have a bug in the log collection - I'll figure that out
20:17:57 dansmith kashyap: and if not, no. we have races that sit dormant for years until something changes in infra, things go in a different order, and we start getting breakage
20:18:01 dansmith kashyap: ack
20:18:33 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: TrivialFix: Add a space between messages https://review.openstack.org/541436
20:18:49 mriedem odyssey4me: ok here was that error you mentioned http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-conductor.log.txt.gz#_2018-02-05_18_19_11_428
20:19:19 odyssey4me mriedem yep, eith the instance error state just a little later
20:21:21 mriedem odyssey4me: i see that nova-manage cell_v2 create_cell didn't exist in newton, so when those commands run, are they against ocata?
20:21:45 odyssey4me yes, in our newton deploy there is no cell management done
20:22:25 kashyap dansmith: Scooping up my final active neurons: You want to see clear _functional test_ evidence that the override of CONF.libvirt.live_migration_bandwidth doesn't happen, yes? (Though, Sahid claims it doesn't.)
20:22:33 odyssey4me mriedem newton only does these: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/newton/tasks/nova_db_setup.yml
20:22:51 dansmith kashyap: no.
20:22:53 mriedem odyssey4me: there are a bunch of errors in the nova-compute logs http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz
20:23:08 dansmith kashyap: I want human analysis of whether or not setting the thing to 1 and 100 in two threads is asking for trouble
20:23:48 dansmith kashyap: as far as I can tell, the migration is started with the CONF value, and then set to 1 to slow things down, but there is no locking or mechanism to ensure the slowdown call is done after the value is set to the one from conf
20:24:04 odyssey4me mriedem hmm, even late in the log, yes: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_449
20:24:04 kashyap (Okay, between the two threads where it is being set. Needs more code reading...)
20:24:21 odyssey4me mriedem actually, that seems to recover: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_469
20:24:28 dansmith kashyap: I can ask someone else, I thought you would be able to dig into the libvirt docs and validate or discredit sahid's assertion that it doesn't matter (which doesn't make sense to me)
20:24:31 kashyap dansmith: Yeah, it is started with the CONF value
20:25:03 dansmith kashyap: right, but I don't think there's anything that ensures that that is set before we make the call to slow it down
20:25:15 mriedem odyssey4me: right it logs a warning on the first time it doesn't find it
20:25:50 dansmith kashyap: so it seems to me like it could be started, not reach the point at which the conf value is used, we set the speed limit, then the conf value gets applied
20:25:50 odyssey4me these are somewhat odd though: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_14_01_860
20:25:55 kashyap dansmith: Okay, you'd want to know how the two values that are being set are coordinated with some locking mechanism with the underlying libvirt API
20:26:12 dansmith kashyap: no. nevermind, go to sleep, I'll try to dig into it
20:26:52 mriedem odyssey4me: ok so in the console output, we see that compute show up in the service list here http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_43_557748
20:27:03 ameeda_ mriedem: Hello :)
20:28:58 mriedem odyssey4me: it might be worthwhile to run discover_hosts with --verbose http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_48_024543
20:29:02 mriedem to see what it dumps out
20:29:40 mriedem odyssey4me: i do remember that with devstack-based CI, we'd have issues in grenade where nova-compute would take awhile to bring up and create the compute_nodes record, and discover_hosts would miss it
20:30:25 mriedem https://review.openstack.org/#/c/488381/
20:30:36 odyssey4me mriedem yeah, that's why we added the wait until a compute agent was found - that seems to have worked for us so far, but perhaps we need more sheps
20:30:39 mriedem however, you have that "openstack compute service list" check
20:31:32 mordred efried: looks like you have a fun debugging task on your plate
20:31:33 odyssey4me but yeah, lemme add the --verbose CLI option when our debug flag is enabled, which would be very useful here
20:31:45 efried mordred: indeeed
20:33:08 mriedem odyssey4me: are the /etc/ files captured?
20:33:28 odyssey4me mriedem unfortunately not - we had to turn that off 'cos we were eating up all the infra inodes :/
20:33:50 odyssey4me we could add that back, but tar.gz them so we eat fewer inodes :)
20:33:58 mriedem ok, i can't tell if the [database]/connection value in nova.conf that the nova-compute is using is the same database connection as the cell1 this creates
20:36:42 mriedem odyssey4me: anyway i think we can see that the compute_nodes entry is created here for the compute host http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_469
20:36:49 mriedem at 18:15:57.469
20:36:58 mriedem it shows up in the service list here:
20:36:58 mriedem http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_43_557748
20:37:09 mriedem "end": "2018-02-05 18:15:43.505982",
20:37:31 odyssey4me mriedem db config template is here, if that helps a bit: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/templates/nova.conf.j2#L213-L233
20:37:56 odyssey4me each db will have a unique name, and each db user/password will also be unique
20:38:17 dansmith kashyap: I explained my concern in more detail, maybe you could dig into docs/code tomorrow with that extra info to validate one way or the other?
20:39:33 mriedem odyssey4me: looks ok
20:39:37 openstackgerrit melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442
20:41:44 dansmith melwitt: nice
20:41:48 mriedem odyssey4me: also throw the --verbose option on create_cell
20:43:18 mriedem ultimately my guess is discover_hosts isn't picking up the compute host and mapping it to the cell
20:43:26 mriedem just not sure why
20:43:43 mriedem could be a timing issue, but that's what the openstack service list check is for
20:44:28 melwitt dansmith: thanks for pointing it out on the original review
20:44:36 dansmith melwitt: yar
20:44:48 efried mriedem: What's your trick for seeing those nova-new results before they're reported to gerrit?
20:44:57 dansmith melwitt: that race is the whole reason I wrote this eventing stuff so I get all sad when it gets broken
20:45:01 odyssey4me yeah, that's my one suspicion... a race condition of some sort - the other is that perhaps there's an online migration required which wasn't executed
20:45:19 mriedem efried: status.openstack.org/zuul
20:45:37 mriedem odyssey4me: i see that the online data migrations are skipped
20:45:37 efried Oh, look, those are links!
20:45:52 odyssey4me mriedem thanks again for looking through all that, and for sanity checking the process to verify that everything looks like it's doing what it should... I'll get a better set of test logs/configs going and we can revisit when those are good
20:46:16 mriedem odyssey4me: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_03_50_709904
20:46:22 mriedem <openstack1> Task "Disable the online migrations requirement" has been omitted from the job because the conditional "[u"inventory_hostname == groups['nova_conductor'][0]", u'not data_migrations | skipped', u'data_migrations | succeeded']" was evaluated as "False"

Earlier   Later