| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 21:09:37 | tonyb | cool | |
| 21:13:35 | sdague | tonyb: yes, also, work upstream :P | |
| 21:23:25 | efried | sdague mriedem Okay, we've figured out how that change affected us. We need to do some stuff to the systemctl service file and restart compute. As of now, we're doing that after devstack finishes, not caring that compute wasn't coming up all the way during the stack. With this change, it started to matter that compute wasn't starting. | |
| 21:23:52 | efried | So we're going to figure out how to tweak the service file before stacking starts. Should make compute come right up during the stack process. | |
| 21:24:28 | mriedem | live migration job is done on my sanity check change http://logs.openstack.org/87/488187/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/482b6f4/ | |
| 21:27:20 | mriedem | oh i've got a bug in there | |
| 21:33:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check delete_allocation_for_instance https://review.openstack.org/488187 | |
| 21:35:45 | dansmith | jaypipes: https://bugs.launchpad.net/nova/+bug/1707071 | |
| 21:35:46 | openstack | Launchpad bug 1707071 in OpenStack Compute (nova) "Compute nodes will fight over allocations during migration" [Undecided,New] | |
| 21:35:52 | dansmith | dude how awesome is that bug number? | |
| 21:36:03 | dansmith | palindromic and all primes | |
| 21:37:49 | dansmith | well, I guess zero isn't a prime.. damn | |
| 21:40:37 | mriedem | hmm, so on rebuild to the same host | |
| 21:40:54 | mriedem | conductor calls select_destinations | |
| 21:41:11 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L770 | |
| 21:41:23 | mriedem | will we double allocate that host then? | |
| 21:43:27 | mriedem | oh nvm | |
| 21:43:29 | mriedem | that's if not host | |
| 21:44:51 | mriedem | whew | |
| 21:47:44 | smcginnis | dansmith: You'll have to try to catch 1737371 | |
| 21:52:15 | dansmith | smcginnis: yeah | |
| 21:56:34 | melwitt | time to buy a lotto ticket | |
| 22:07:58 | mriedem | jaypipes: dansmith: ok, https://review.openstack.org/#/c/483566/ | |
| 22:08:06 | mriedem | there is one thing in there that worries me | |
| 22:08:19 | mriedem | https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@171 | |
| 22:09:31 | dansmith | hrm | |
| 22:10:13 | mriedem | if len(selected_hosts) < num_instances: | |
| 22:10:13 | mriedem | oh L86 | |
| 22:10:42 | dansmith | what happens if we return less than enough hosts to whatever calls us? | |
| 22:11:00 | dansmith | heh yeah that | |
| 22:11:30 | dansmith | and min/max_instances is only a quota check, so no problem there | |
| 22:12:29 | mriedem | ok +2 | |
| 22:12:35 | mriedem | i tried my hardest | |
| 22:12:39 | mriedem | to find fault | |
| 22:14:08 | dansmith | ack | |
| 22:14:31 | melwitt | mriedem: docstring doesn't match params here https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@252 if you wanted something :P | |
| 22:14:37 | mriedem | there might still be something to https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@208 | |
| 22:14:59 | mriedem | where if we know allocation requests are constantly failing for a host, we should stop trying it | |
| 22:15:17 | mriedem | omg | |
| 22:15:19 | mriedem | -10 | |
| 22:15:36 | melwitt | hehe | |
| 22:15:38 | mriedem | you know i think i was looking for where cn_uuid was used in there too | |
| 22:15:39 | mriedem | b/c of the docstring | |
| 22:18:53 | mriedem | pike-3 tag https://review.openstack.org/#/c/488218/ | |
| 22:21:09 | mriedem | jaypipes: thanks for hanging in there | |
| 22:21:24 | mriedem | ooo just in time as laura rolls back up to the house with the kid | |
| 22:21:37 | mriedem | time to get my county fair food eating clothes on | |
| 22:21:43 | dansmith | heh | |
| 22:21:52 | mriedem | over sized drawers, bib, etc | |
| 22:24:41 | mriedem | sdague: the compute host wait thing also blew up the cellsv1 job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-cells-ubuntu-xenial/8ac25f1/logs/devstacklog.txt.gz | |
| 22:28:03 | mriedem | dansmith: jaypipes: here is the sanity check in action on the live migration job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/107a810/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_27_22_21_14_766229 | |
| 22:28:26 | mriedem | i didn't see any cases where the source node deleting the allocation isn't in the list of current allocations | |
| 22:28:36 | mriedem | but it's definitely stomping over the 'double up' allocatoins | |
| 22:29:04 | mriedem | Removing allocations for instance which are currently against more than one compute node resource provider. Current allocations: {u'13b1e5e0-66ef-4533-9a07-b1a3220d6b00': {u'generation': 8, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, u'7aa9619d-db83-4da9-b822-f4d66e7143f8': {u'generation': 6, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}} | |
| 22:33:20 | dansmith | okay I think that's just getting lucky, | |
| 22:33:29 | dansmith | as I think it could either win or lose that race, but cool | |
| 22:37:45 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 22:37:45 | mriedem | oh no you didn't | |
| 22:44:32 | melwitt | dunno | |
| 22:44:54 | mikal | I'm going to be a bad man and reply to him cc'ing openstack-dev I suppose | |
| 22:45:06 | mikal | I don't want to exclude the nova team from the conversation | |
| 23:35:57 | tonyb | mikal: Yeah the only email was the private one but you've seen the conversation here and on the review | |
| 23:36:54 | tonyb | mikal: I don't see you reply did you send it already? | |
| 23:43:02 | mikal | tonyb: I cc'ed you and sent it to -dev | |
| 23:43:26 | mikal | Subject: Re: [Nova] Broken nova-lxd | |
| 23:44:00 | mikal | http://lists.openstack.org/pipermail/openstack-dev/2017-July/120259.html | |
| 23:44:01 | tonyb | mikal: okay I don't see it yet probably stuck in a mail queue somewhere | |
| 23:50:47 | openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | |
| #openstack-nova - 2017-07-28 | |||
| 00:26:52 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 00:26:53 | openstackgerrit | Dan Smith proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | |
| 00:55:46 | jgriffith | mriedem_afk smcginnis ildikov I guess I'll have to accept defeat :( | |
| 00:56:34 | jgriffith | mriedem_afk smcginnis ildikov I updated the comments in the review but there's something bigger going on it looks like, inparticular instances going to error state and libvirt failing to clean them up | |
| 00:56:36 | jgriffith | http://logs.openstack.org/85/330285/106/check/gate-tempest-dsvm-py35-ubuntu-xenial/40072bb/logs/screen-n-cpu.txt.gz?level=TRACE#_Jul_26_09_34_33_062305 | |
| 00:56:48 | openstackgerrit | Dan Smith proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 00:56:48 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 00:57:50 | jgriffith | mriedem_afk smcginnis ildikov the compute host not found error is kinda concerning, wondering if we need a rebase again? | |
| 00:57:52 | jgriffith | anyway | |
| 00:58:00 | jgriffith | I'm heading home, catch ya'll later | |
| 01:53:41 | zul | mikal: I sent a patch to unbreak nova-lxd btw | |
| 01:53:48 | dims | mriedem_afk : which devstack-gate change reset stuff today? https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+status:merged | |
| 01:54:36 | dims | the "Set up stable/pike" one? mriedem_afk | |
| 01:59:41 | dansmith | dims: no, just a sec | |
| 01:59:57 | dansmith | dims: it was devstack, and this is the revert: https://review.openstack.org/#/c/488201/ | |
| 02:00:00 | dansmith | which has a link to the original | |
| 02:00:22 | dims | gotcha. thanks dansmith | |
| 02:00:34 | dansmith | dims: that merged and we're waiting for the two nova patches that got kicked out to merge before I update the tag | |
| 02:01:09 | dansmith | dims: this and the one below: https://review.openstack.org/#/c/483566 | |
| 02:03:07 | dims | dansmith : looks like it will take another 2 hours at least | |
| 02:03:13 | dansmith | dims: at least | |
| 02:03:28 | dansmith | dims: I'm watching it though and will update the tag commit when it lands | |
| 02:03:59 | dansmith | a thrilling thursday evening | |
| 02:04:06 | dims | sounds good dansmith. i'll take care of it when i wake up if ttx hasn't already | |
| 02:04:16 | dansmith | cool, thanks | |
| 02:58:51 | smcginnis | jgriffith: Thanks for digging in to it. Maybe collect notes in an etherpad or something to see if anyone else has more ideas? | |
| 03:17:39 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update URLs in docs, comments and setup.cfg https://review.openstack.org/488271 | |
| 03:19:26 | jgriffith | smcginnis roger dodger | |