| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 21:41:23 | mriedem | will we double allocate that host then? | |
| 21:43:27 | mriedem | oh nvm | |
| 21:43:29 | mriedem | that's if not host | |
| 21:44:51 | mriedem | whew | |
| 21:47:44 | smcginnis | dansmith: You'll have to try to catch 1737371 | |
| 21:52:15 | dansmith | smcginnis: yeah | |
| 21:56:34 | melwitt | time to buy a lotto ticket | |
| 22:07:58 | mriedem | jaypipes: dansmith: ok, https://review.openstack.org/#/c/483566/ | |
| 22:08:06 | mriedem | there is one thing in there that worries me | |
| 22:08:19 | mriedem | https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@171 | |
| 22:09:31 | dansmith | hrm | |
| 22:10:13 | mriedem | oh L86 | |
| 22:10:13 | mriedem | if len(selected_hosts) < num_instances: | |
| 22:10:42 | dansmith | what happens if we return less than enough hosts to whatever calls us? | |
| 22:11:00 | dansmith | heh yeah that | |
| 22:11:30 | dansmith | and min/max_instances is only a quota check, so no problem there | |
| 22:12:29 | mriedem | ok +2 | |
| 22:12:35 | mriedem | i tried my hardest | |
| 22:12:39 | mriedem | to find fault | |
| 22:14:08 | dansmith | ack | |
| 22:14:31 | melwitt | mriedem: docstring doesn't match params here https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@252 if you wanted something :P | |
| 22:14:37 | mriedem | there might still be something to https://review.openstack.org/#/c/483566/20/nova/scheduler/filter_scheduler.py@208 | |
| 22:14:59 | mriedem | where if we know allocation requests are constantly failing for a host, we should stop trying it | |
| 22:15:17 | mriedem | omg | |
| 22:15:19 | mriedem | -10 | |
| 22:15:36 | melwitt | hehe | |
| 22:15:38 | mriedem | you know i think i was looking for where cn_uuid was used in there too | |
| 22:15:39 | mriedem | b/c of the docstring | |
| 22:18:53 | mriedem | pike-3 tag https://review.openstack.org/#/c/488218/ | |
| 22:21:09 | mriedem | jaypipes: thanks for hanging in there | |
| 22:21:24 | mriedem | ooo just in time as laura rolls back up to the house with the kid | |
| 22:21:37 | mriedem | time to get my county fair food eating clothes on | |
| 22:21:43 | dansmith | heh | |
| 22:21:52 | mriedem | over sized drawers, bib, etc | |
| 22:24:41 | mriedem | sdague: the compute host wait thing also blew up the cellsv1 job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-cells-ubuntu-xenial/8ac25f1/logs/devstacklog.txt.gz | |
| 22:28:03 | mriedem | dansmith: jaypipes: here is the sanity check in action on the live migration job http://logs.openstack.org/87/488187/2/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/107a810/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_27_22_21_14_766229 | |
| 22:28:26 | mriedem | i didn't see any cases where the source node deleting the allocation isn't in the list of current allocations | |
| 22:28:36 | mriedem | but it's definitely stomping over the 'double up' allocatoins | |
| 22:29:04 | mriedem | Removing allocations for instance which are currently against more than one compute node resource provider. Current allocations: {u'13b1e5e0-66ef-4533-9a07-b1a3220d6b00': {u'generation': 8, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}, u'7aa9619d-db83-4da9-b822-f4d66e7143f8': {u'generation': 6, u'resources': {u'VCPU': 1, u'MEMORY_MB': 64}}} | |
| 22:33:20 | dansmith | okay I think that's just getting lucky, | |
| 22:33:29 | dansmith | as I think it could either win or lose that race, but cool | |
| 22:37:45 | mriedem | oh no you didn't | |
| 22:37:45 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 22:44:32 | melwitt | dunno | |
| 22:44:54 | mikal | I'm going to be a bad man and reply to him cc'ing openstack-dev I suppose | |
| 22:45:06 | mikal | I don't want to exclude the nova team from the conversation | |
| 23:35:57 | tonyb | mikal: Yeah the only email was the private one but you've seen the conversation here and on the review | |
| 23:36:54 | tonyb | mikal: I don't see you reply did you send it already? | |
| 23:43:02 | mikal | tonyb: I cc'ed you and sent it to -dev | |
| 23:43:26 | mikal | Subject: Re: [Nova] Broken nova-lxd | |
| 23:44:00 | mikal | http://lists.openstack.org/pipermail/openstack-dev/2017-July/120259.html | |
| 23:44:01 | tonyb | mikal: okay I don't see it yet probably stuck in a mail queue somewhere | |
| 23:50:47 | openstackgerrit | Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/488125 | |
| #openstack-nova - 2017-07-28 | |||
| 00:26:52 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 00:26:53 | openstackgerrit | Dan Smith proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | |
| 00:36:26 | openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | |
| 00:36:27 | openstackgerrit | Michael Still proposed openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831 | |
| 00:55:46 | jgriffith | mriedem_afk smcginnis ildikov I guess I'll have to accept defeat :( | |
| 00:56:34 | jgriffith | mriedem_afk smcginnis ildikov I updated the comments in the review but there's something bigger going on it looks like, inparticular instances going to error state and libvirt failing to clean them up | |
| 00:56:36 | jgriffith | http://logs.openstack.org/85/330285/106/check/gate-tempest-dsvm-py35-ubuntu-xenial/40072bb/logs/screen-n-cpu.txt.gz?level=TRACE#_Jul_26_09_34_33_062305 | |
| 00:56:48 | openstackgerrit | Dan Smith proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 00:56:48 | openstackgerrit | Dan Smith proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 00:57:50 | jgriffith | mriedem_afk smcginnis ildikov the compute host not found error is kinda concerning, wondering if we need a rebase again? | |
| 00:57:52 | jgriffith | anyway | |
| 00:58:00 | jgriffith | I'm heading home, catch ya'll later | |
| 01:53:41 | zul | mikal: I sent a patch to unbreak nova-lxd btw | |
| 01:53:48 | dims | mriedem_afk : which devstack-gate change reset stuff today? https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+status:merged | |
| 01:54:36 | dims | the "Set up stable/pike" one? mriedem_afk | |
| 01:59:41 | dansmith | dims: no, just a sec | |
| 01:59:57 | dansmith | dims: it was devstack, and this is the revert: https://review.openstack.org/#/c/488201/ | |
| 02:00:00 | dansmith | which has a link to the original | |
| 02:00:22 | dims | gotcha. thanks dansmith | |
| 02:00:34 | dansmith | dims: that merged and we're waiting for the two nova patches that got kicked out to merge before I update the tag | |
| 02:01:09 | dansmith | dims: this and the one below: https://review.openstack.org/#/c/483566 | |
| 02:03:07 | dims | dansmith : looks like it will take another 2 hours at least | |
| 02:03:13 | dansmith | dims: at least | |
| 02:03:28 | dansmith | dims: I'm watching it though and will update the tag commit when it lands | |
| 02:03:59 | dansmith | a thrilling thursday evening | |
| 02:04:06 | dims | sounds good dansmith. i'll take care of it when i wake up if ttx hasn't already | |
| 02:04:16 | dansmith | cool, thanks | |
| 02:58:51 | smcginnis | jgriffith: Thanks for digging in to it. Maybe collect notes in an etherpad or something to see if anyone else has more ideas? | |
| 03:17:39 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update URLs in docs, comments and setup.cfg https://review.openstack.org/488271 | |
| 03:19:26 | jgriffith | smcginnis roger dodger | |
| 03:29:02 | mriedem_afk | jgriffith: the compute host not found warnings at the start of the n-cpu logs are 'normal' right now | |
| 03:29:28 | jgriffith | mriedem noted | |
| 03:42:55 | mriedem | dansmith: #2 in the gate, i can take over if you want | |
| 03:43:08 | dansmith | I'm watchin' | |
| 03:43:21 | dansmith | you got to sleep | |
| 03:43:22 | dansmith | s/got/go/ | |
| 03:44:09 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Remove run_tests.sh https://review.openstack.org/488276 | |
| 03:53:40 | mriedem | i don't sleep | |
| 03:54:53 | dansmith | hah | |
| 03:59:29 | dansmith | 10 minutes | |
| 03:59:57 | dansmith | ooh, 1 minute | |
| 04:01:40 | jgriffith | mriedem smcginnis https://etherpad.openstack.org/p/cinder-v3-attach-nova-traces | |
| 04:02:04 | jgriffith | I can add more as I go here and maybe even send a request out to the ML tomorrow if there might be anybody interested | |
| 04:02:24 | jgriffith | My brain has turned a bit mushy after waking through all the bdm stuff all day over and over | |
| 04:03:47 | jgriffith | One thing I left out... I have a system I reproduced this on that the /dev/sdd device is in fact there and the connection is still valid, but that periodic query against it still fails... I didn't go down that rabbit hole yet though | |